Nov 02 17:26:53.396137 ip-10-0-142-26 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 02 17:26:53.396148 ip-10-0-142-26 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 02 17:26:53.396155 ip-10-0-142-26 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 02 17:26:53.396404 ip-10-0-142-26 systemd[1]: Failed to start Kubernetes Kubelet. Nov 02 17:27:03.645798 ip-10-0-142-26 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 02 17:27:03.645813 ip-10-0-142-26 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot cc2062ece693427d8b0ca65bb170ce11 -- Nov 02 17:29:01.821567 ip-10-0-142-26 systemd[1]: Starting Kubernetes Kubelet... Nov 02 17:29:02.217417 ip-10-0-142-26 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:29:02.217417 ip-10-0-142-26 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 17:29:02.217417 ip-10-0-142-26 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:29:02.217417 ip-10-0-142-26 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:29:02.221720 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221637 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:29:02.221720 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221720 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:29:02.221792 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221726 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:29:02.221792 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221730 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:29:02.221792 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221733 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:29:02.221792 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221736 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:29:02.221792 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221740 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:29:02.221792 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221743 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:29:02.221792 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221747 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:29:02.221792 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221749 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:29:02.221792 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221752 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:29:02.221792 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221755 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:29:02.221792 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221758 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:29:02.221792 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221760 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:29:02.221792 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221763 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:29:02.221792 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221766 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:29:02.221792 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221768 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:29:02.221792 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221771 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:29:02.221792 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221773 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:29:02.221792 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221776 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:29:02.221792 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221779 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:29:02.222234 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221782 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:29:02.222234 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221784 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:29:02.222234 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221787 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:29:02.222234 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221790 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:29:02.222234 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221796 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:29:02.222234 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221799 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:29:02.222234 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221802 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:29:02.222234 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221805 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:29:02.222234 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221808 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:29:02.222234 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221811 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:29:02.222234 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221813 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:29:02.222234 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221816 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:29:02.222234 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221819 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:29:02.222234 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221822 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:29:02.222234 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221824 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:29:02.222234 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221827 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:29:02.222234 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221829 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:29:02.222234 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221832 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:29:02.222234 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221835 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:29:02.222234 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221837 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:29:02.222710 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221842 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:29:02.222710 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221845 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:29:02.222710 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221848 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:29:02.222710 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221851 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:29:02.222710 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221853 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:29:02.222710 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221856 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:29:02.222710 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221858 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:29:02.222710 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221861 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:29:02.222710 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221864 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:29:02.222710 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221866 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:29:02.222710 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221869 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:29:02.222710 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221871 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:29:02.222710 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221874 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:29:02.222710 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221878 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:29:02.222710 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221880 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:29:02.222710 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221883 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:29:02.222710 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221886 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:29:02.222710 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221889 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:29:02.222710 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221892 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:29:02.223182 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221894 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:29:02.223182 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221897 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:29:02.223182 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221900 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:29:02.223182 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221902 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:29:02.223182 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221905 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:29:02.223182 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221907 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:29:02.223182 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221910 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:29:02.223182 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221913 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:29:02.223182 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221915 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:29:02.223182 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221918 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:29:02.223182 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221920 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:29:02.223182 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221923 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:29:02.223182 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221925 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:29:02.223182 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221928 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:29:02.223182 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221930 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:29:02.223182 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221933 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:29:02.223182 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221935 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:29:02.223182 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221937 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:29:02.223182 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221940 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:29:02.223182 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221943 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:29:02.223678 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221946 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:29:02.223678 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221948 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:29:02.223678 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221951 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:29:02.223678 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221954 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:29:02.223678 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221956 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:29:02.223678 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221959 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:29:02.223678 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221962 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:29:02.223678 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221965 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:29:02.223678 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221968 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:29:02.223678 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221971 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:29:02.223678 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221973 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:29:02.223678 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221977 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:29:02.223678 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221979 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:29:02.223678 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221982 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:29:02.223678 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221984 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:29:02.223678 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221987 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:29:02.223678 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221989 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:29:02.223678 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221992 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:29:02.223678 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221994 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:29:02.223678 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.221997 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:29:02.224172 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222000 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:29:02.224172 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222003 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:29:02.224172 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222005 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:29:02.224172 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222008 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:29:02.224172 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222010 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:29:02.224172 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222013 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:29:02.224172 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222892 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:29:02.224172 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222900 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:29:02.224172 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222903 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:29:02.224172 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222906 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:29:02.224172 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222909 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:29:02.224172 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222911 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:29:02.224172 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222914 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:29:02.224172 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222917 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:29:02.224172 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222919 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:29:02.224172 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222922 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:29:02.224172 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222927 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:29:02.224172 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222939 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:29:02.224172 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222943 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:29:02.224172 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222946 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:29:02.224716 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222949 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:29:02.224716 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222951 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:29:02.224716 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222954 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:29:02.224716 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222956 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:29:02.224716 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222965 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:29:02.224716 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222968 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:29:02.224716 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222971 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:29:02.224716 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222974 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:29:02.224716 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222976 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:29:02.224716 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222979 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:29:02.224716 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222982 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:29:02.224716 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222985 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:29:02.224716 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222987 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:29:02.224716 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222990 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:29:02.224716 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.222992 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:29:02.224716 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223001 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:29:02.224716 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223004 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:29:02.224716 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223006 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:29:02.224716 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223009 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:29:02.224716 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223012 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:29:02.225187 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223015 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:29:02.225187 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223017 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:29:02.225187 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223020 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:29:02.225187 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223022 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:29:02.225187 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223025 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:29:02.225187 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223027 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:29:02.225187 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223030 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:29:02.225187 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223033 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:29:02.225187 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223035 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:29:02.225187 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223038 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:29:02.225187 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223041 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:29:02.225187 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223044 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:29:02.225187 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223047 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:29:02.225187 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223049 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:29:02.225187 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223052 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:29:02.225187 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223054 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:29:02.225187 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223057 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:29:02.225187 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223066 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:29:02.225187 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223069 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:29:02.225647 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223072 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:29:02.225647 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223074 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:29:02.225647 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223077 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:29:02.225647 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223079 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:29:02.225647 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223082 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:29:02.225647 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223085 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:29:02.225647 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223087 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:29:02.225647 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223095 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:29:02.225647 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223097 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:29:02.225647 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223100 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:29:02.225647 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223102 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:29:02.225647 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223105 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:29:02.225647 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223107 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:29:02.225647 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223110 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:29:02.225647 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223113 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:29:02.225647 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223115 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:29:02.225647 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223118 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:29:02.225647 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223120 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:29:02.225647 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223123 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:29:02.226093 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223126 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:29:02.226093 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223128 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:29:02.226093 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223131 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:29:02.226093 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223133 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:29:02.226093 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223139 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:29:02.226093 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223142 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:29:02.226093 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223144 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:29:02.226093 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223147 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:29:02.226093 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223149 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:29:02.226093 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223152 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:29:02.226093 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223155 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:29:02.226093 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223157 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:29:02.226093 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223166 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:29:02.226093 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223169 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:29:02.226093 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223172 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:29:02.226093 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223174 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:29:02.226093 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223177 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:29:02.226093 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223179 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:29:02.226093 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223182 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:29:02.226093 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223185 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:29:02.226576 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223187 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:29:02.226576 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223190 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:29:02.226576 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223192 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:29:02.226576 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223195 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:29:02.226576 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223199 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:29:02.226576 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223202 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:29:02.226576 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223205 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:29:02.226576 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223207 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:29:02.226576 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223210 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:29:02.226576 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223214 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:29:02.226576 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223217 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:29:02.226576 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223219 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:29:02.226576 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223222 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:29:02.226576 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223224 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:29:02.226576 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223782 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:29:02.226576 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223788 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:29:02.226576 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223793 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:29:02.226576 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223796 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:29:02.226576 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223798 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:29:02.227026 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223801 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:29:02.227026 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223803 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:29:02.227026 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223806 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:29:02.227026 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223809 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:29:02.227026 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223811 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:29:02.227026 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223820 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:29:02.227026 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223822 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:29:02.227026 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223825 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:29:02.227026 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223828 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:29:02.227026 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223830 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:29:02.227026 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223833 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:29:02.227026 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223835 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:29:02.227026 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223838 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:29:02.227026 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223840 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:29:02.227026 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223843 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:29:02.227026 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223845 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:29:02.227026 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223848 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:29:02.227026 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223850 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:29:02.227026 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223852 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:29:02.227485 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223855 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:29:02.227485 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223857 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:29:02.227485 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223860 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:29:02.227485 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223863 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:29:02.227485 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223865 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:29:02.227485 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223868 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:29:02.227485 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223871 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:29:02.227485 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223873 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:29:02.227485 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223876 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:29:02.227485 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223878 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:29:02.227485 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223882 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:29:02.227485 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223885 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:29:02.227485 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223887 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:29:02.227485 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223890 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:29:02.227485 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223893 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:29:02.227485 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223895 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:29:02.227485 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223898 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:29:02.227485 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223901 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:29:02.227485 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223903 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:29:02.227485 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223911 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:29:02.227951 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223914 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:29:02.227951 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223917 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:29:02.227951 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223919 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:29:02.227951 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223922 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:29:02.227951 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223924 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:29:02.227951 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223927 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:29:02.227951 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223929 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:29:02.227951 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223932 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:29:02.227951 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223935 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:29:02.227951 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223939 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:29:02.227951 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223943 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:29:02.227951 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223946 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:29:02.227951 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223949 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:29:02.227951 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223953 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:29:02.227951 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223961 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:29:02.227951 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223964 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:29:02.227951 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223967 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:29:02.227951 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223969 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:29:02.227951 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223972 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:29:02.228415 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223974 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:29:02.228415 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223977 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:29:02.228415 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223979 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:29:02.228415 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223983 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:29:02.228415 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223985 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:29:02.228415 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223988 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:29:02.228415 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223990 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:29:02.228415 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223993 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:29:02.228415 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223996 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:29:02.228415 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.223998 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:29:02.228415 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.224000 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:29:02.228415 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.224009 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:29:02.228415 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.224012 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:29:02.228415 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.224020 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:29:02.228415 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.224033 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:29:02.228415 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.224036 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:29:02.228415 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.224038 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:29:02.228415 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.224041 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:29:02.228415 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.224044 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:29:02.228415 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.224047 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:29:02.228898 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.224051 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:29:02.228898 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.224054 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:29:02.228898 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.224056 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:29:02.228898 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.224059 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:29:02.228898 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.224061 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:29:02.228898 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.224064 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:29:02.228898 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.224066 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:29:02.228898 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.224069 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:29:02.228898 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.224072 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:29:02.228898 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.224075 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:29:02.228898 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.224077 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:29:02.228898 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.224080 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:29:02.228898 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.224083 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:29:02.228898 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.224085 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:29:02.228898 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.224088 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:29:02.228898 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.224092 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:29:02.228898 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.224094 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:29:02.228898 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.224097 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:29:02.228898 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.224099 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:29:02.228898 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.224102 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:29:02.229390 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.224105 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:29:02.229390 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.224107 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:29:02.229390 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.224111 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:29:02.229390 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225029 2580 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 17:29:02.229390 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225039 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 17:29:02.229390 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225049 2580 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 17:29:02.229390 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225060 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 17:29:02.229390 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225065 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 17:29:02.229390 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225069 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 17:29:02.229390 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225073 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 17:29:02.229390 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225078 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 17:29:02.229390 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225081 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 17:29:02.229390 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225084 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 17:29:02.229390 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225087 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 17:29:02.229390 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225091 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 17:29:02.229390 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225094 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 17:29:02.229390 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225097 2580 flags.go:64] FLAG: --cgroup-root="" Nov 02 17:29:02.229390 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225100 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 17:29:02.229390 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225103 2580 flags.go:64] FLAG: --client-ca-file="" Nov 02 17:29:02.229390 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225106 2580 flags.go:64] FLAG: --cloud-provider="external" Nov 02 17:29:02.229390 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225111 2580 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 17:29:02.229390 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225118 2580 flags.go:64] FLAG: --cluster-domain="" Nov 02 17:29:02.229390 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225121 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 17:29:02.229390 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225124 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 17:29:02.229951 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225127 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 17:29:02.229951 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225131 2580 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 17:29:02.229951 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225135 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 17:29:02.229951 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225139 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 17:29:02.229951 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225142 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 17:29:02.229951 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225145 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 17:29:02.229951 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225148 2580 flags.go:64] FLAG: --contention-profiling="false" Nov 02 17:29:02.229951 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225151 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 17:29:02.229951 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225154 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 17:29:02.229951 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225157 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 17:29:02.229951 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225160 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 17:29:02.229951 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225165 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 17:29:02.229951 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225168 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 17:29:02.229951 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225171 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 17:29:02.229951 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225174 2580 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 17:29:02.229951 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225184 2580 flags.go:64] FLAG: --enable-server="true" Nov 02 17:29:02.229951 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225187 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 17:29:02.229951 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225194 2580 flags.go:64] FLAG: --event-burst="100" Nov 02 17:29:02.229951 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225198 2580 flags.go:64] FLAG: --event-qps="50" Nov 02 17:29:02.229951 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225201 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 17:29:02.229951 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225204 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 17:29:02.229951 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225207 2580 flags.go:64] FLAG: --eviction-hard="" Nov 02 17:29:02.229951 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225211 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 17:29:02.229951 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225214 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 17:29:02.230547 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225218 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 17:29:02.230547 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225221 2580 flags.go:64] FLAG: --eviction-soft="" Nov 02 17:29:02.230547 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225224 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 17:29:02.230547 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225227 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 17:29:02.230547 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225231 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 17:29:02.230547 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225234 2580 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 17:29:02.230547 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225237 2580 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 17:29:02.230547 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225240 2580 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 17:29:02.230547 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225243 2580 flags.go:64] FLAG: --feature-gates="" Nov 02 17:29:02.230547 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225247 2580 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 17:29:02.230547 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225250 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 17:29:02.230547 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225254 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 17:29:02.230547 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225257 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 17:29:02.230547 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225260 2580 flags.go:64] FLAG: --healthz-port="10248" Nov 02 17:29:02.230547 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225263 2580 flags.go:64] FLAG: --help="false" Nov 02 17:29:02.230547 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225266 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-142-26.ec2.internal" Nov 02 17:29:02.230547 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225269 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 17:29:02.230547 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225272 2580 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 17:29:02.230547 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225276 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 17:29:02.230547 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225279 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 17:29:02.230547 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225282 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 17:29:02.230547 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225285 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 17:29:02.230547 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225288 2580 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 17:29:02.231100 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225291 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 17:29:02.231100 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225316 2580 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 17:29:02.231100 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225319 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 17:29:02.231100 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225323 2580 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 17:29:02.231100 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225325 2580 flags.go:64] FLAG: --kube-reserved="" Nov 02 17:29:02.231100 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225328 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 17:29:02.231100 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225331 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 17:29:02.231100 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225335 2580 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 17:29:02.231100 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225338 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 17:29:02.231100 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225340 2580 flags.go:64] FLAG: --lock-file="" Nov 02 17:29:02.231100 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225343 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 17:29:02.231100 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225346 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 17:29:02.231100 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225349 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 17:29:02.231100 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225362 2580 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 17:29:02.231100 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225365 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 17:29:02.231100 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225369 2580 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 17:29:02.231100 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225372 2580 flags.go:64] FLAG: --logging-format="text" Nov 02 17:29:02.231100 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225375 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 17:29:02.231100 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225378 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 17:29:02.231100 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225381 2580 flags.go:64] FLAG: --manifest-url="" Nov 02 17:29:02.231100 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225384 2580 flags.go:64] FLAG: --manifest-url-header="" Nov 02 17:29:02.231100 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225389 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 17:29:02.231100 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225392 2580 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 17:29:02.231100 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225396 2580 flags.go:64] FLAG: --max-pods="110" Nov 02 17:29:02.231100 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225399 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 17:29:02.231737 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225401 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 17:29:02.231737 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225404 2580 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 17:29:02.231737 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225407 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 17:29:02.231737 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225410 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 17:29:02.231737 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225413 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 17:29:02.231737 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225416 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 17:29:02.231737 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225423 2580 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 17:29:02.231737 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225426 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 17:29:02.231737 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225429 2580 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 17:29:02.231737 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225438 2580 flags.go:64] FLAG: --pod-cidr="" Nov 02 17:29:02.231737 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225441 2580 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 17:29:02.231737 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225444 2580 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 17:29:02.231737 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225447 2580 flags.go:64] FLAG: --pods-per-core="0" Nov 02 17:29:02.231737 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225450 2580 flags.go:64] FLAG: --port="10250" Nov 02 17:29:02.231737 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225454 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 17:29:02.231737 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225456 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0de57b7ca78057d31" Nov 02 17:29:02.231737 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225459 2580 flags.go:64] FLAG: --qos-reserved="" Nov 02 17:29:02.231737 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225462 2580 flags.go:64] FLAG: --read-only-port="10255" Nov 02 17:29:02.231737 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225465 2580 flags.go:64] FLAG: --register-node="true" Nov 02 17:29:02.231737 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225469 2580 flags.go:64] FLAG: --register-with-taints="" Nov 02 17:29:02.231737 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225474 2580 flags.go:64] FLAG: --registry-burst="10" Nov 02 17:29:02.231737 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225477 2580 flags.go:64] FLAG: --registry-qps="5" Nov 02 17:29:02.231737 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225479 2580 flags.go:64] FLAG: --reserved-cpus="" Nov 02 17:29:02.231737 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225482 2580 flags.go:64] FLAG: --reserved-memory="" Nov 02 17:29:02.231737 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225487 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 17:29:02.231737 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225490 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 17:29:02.232404 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225493 2580 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 17:29:02.232404 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225496 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 17:29:02.232404 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225499 2580 flags.go:64] FLAG: --runonce="false" Nov 02 17:29:02.232404 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225501 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 17:29:02.232404 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225505 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 17:29:02.232404 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225508 2580 flags.go:64] FLAG: --seccomp-default="false" Nov 02 17:29:02.232404 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225511 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 17:29:02.232404 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225514 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 17:29:02.232404 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225517 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 17:29:02.232404 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225520 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 17:29:02.232404 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225523 2580 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 17:29:02.232404 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225526 2580 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 17:29:02.232404 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225529 2580 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 17:29:02.232404 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225532 2580 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 17:29:02.232404 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225535 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 17:29:02.232404 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225538 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 17:29:02.232404 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225546 2580 flags.go:64] FLAG: --system-cgroups="" Nov 02 17:29:02.232404 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225549 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 17:29:02.232404 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225555 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 17:29:02.232404 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225558 2580 flags.go:64] FLAG: --tls-cert-file="" Nov 02 17:29:02.232404 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225561 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 17:29:02.232404 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225570 2580 flags.go:64] FLAG: --tls-min-version="" Nov 02 17:29:02.232404 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225578 2580 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 17:29:02.232404 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225582 2580 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 17:29:02.232404 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225585 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 17:29:02.233069 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225588 2580 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 17:29:02.233069 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225592 2580 flags.go:64] FLAG: --v="2" Nov 02 17:29:02.233069 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225596 2580 flags.go:64] FLAG: --version="false" Nov 02 17:29:02.233069 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225600 2580 flags.go:64] FLAG: --vmodule="" Nov 02 17:29:02.233069 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225605 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 17:29:02.233069 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.225608 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 17:29:02.233069 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225796 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:29:02.233069 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225800 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:29:02.233069 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225804 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:29:02.233069 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225807 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:29:02.233069 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225810 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:29:02.233069 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225812 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:29:02.233069 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225815 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:29:02.233069 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225817 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:29:02.233069 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225820 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:29:02.233069 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225822 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:29:02.233069 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225825 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:29:02.233069 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225827 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:29:02.233069 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225831 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:29:02.233069 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225834 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:29:02.233069 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225836 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:29:02.233906 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225839 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:29:02.233906 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225842 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:29:02.233906 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225846 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:29:02.233906 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225855 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:29:02.233906 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225858 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:29:02.233906 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225860 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:29:02.233906 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225863 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:29:02.233906 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225866 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:29:02.233906 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225868 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:29:02.233906 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225871 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:29:02.233906 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225873 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:29:02.233906 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225876 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:29:02.233906 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225879 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:29:02.233906 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225882 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:29:02.233906 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225885 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:29:02.233906 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225887 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:29:02.233906 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225890 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:29:02.233906 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225892 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:29:02.233906 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225895 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:29:02.234534 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225899 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:29:02.234534 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225901 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:29:02.234534 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225904 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:29:02.234534 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225906 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:29:02.234534 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225909 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:29:02.234534 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225912 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:29:02.234534 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225914 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:29:02.234534 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225916 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:29:02.234534 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225919 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:29:02.234534 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225921 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:29:02.234534 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225924 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:29:02.234534 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225926 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:29:02.234534 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225929 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:29:02.234534 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225931 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:29:02.234534 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225934 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:29:02.234534 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225936 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:29:02.234534 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225939 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:29:02.234534 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225947 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:29:02.234534 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225949 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:29:02.234534 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225952 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:29:02.235199 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225956 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:29:02.235199 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225960 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:29:02.235199 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225963 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:29:02.235199 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225965 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:29:02.235199 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225968 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:29:02.235199 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225970 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:29:02.235199 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225973 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:29:02.235199 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225975 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:29:02.235199 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225978 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:29:02.235199 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225980 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:29:02.235199 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225983 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:29:02.235199 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225985 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:29:02.235199 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225989 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:29:02.235199 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225992 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:29:02.235199 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225994 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:29:02.235199 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225997 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:29:02.235199 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.225999 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:29:02.235199 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226002 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:29:02.235199 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226004 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:29:02.235199 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226007 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:29:02.235703 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226009 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:29:02.235703 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226012 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:29:02.235703 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226014 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:29:02.235703 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226017 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:29:02.235703 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226019 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:29:02.235703 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226022 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:29:02.235703 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226024 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:29:02.235703 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226027 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:29:02.235703 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226029 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:29:02.235703 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226032 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:29:02.235703 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226040 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:29:02.235703 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226043 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:29:02.235703 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226045 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:29:02.235703 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226048 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:29:02.235703 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226050 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:29:02.235703 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226053 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:29:02.235703 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226055 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:29:02.235703 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226058 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:29:02.235703 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226061 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:29:02.236148 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226063 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:29:02.236148 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226065 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:29:02.236148 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226068 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:29:02.236148 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226070 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:29:02.236148 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226073 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:29:02.236148 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226076 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:29:02.236148 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226079 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:29:02.236148 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226082 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:29:02.236148 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226084 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:29:02.236148 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226087 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:29:02.236148 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226089 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:29:02.236148 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226092 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:29:02.236148 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226094 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:29:02.236148 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.226120 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:29:02.236148 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226168 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:29:02.236148 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226172 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:29:02.236656 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226175 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:29:02.236656 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226178 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:29:02.236656 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226180 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:29:02.236656 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226183 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:29:02.236656 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226186 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:29:02.236656 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226189 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:29:02.236656 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226191 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:29:02.236656 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226194 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:29:02.236656 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226202 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:29:02.236656 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226206 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:29:02.236656 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226209 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:29:02.236656 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226211 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:29:02.236656 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226214 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:29:02.236656 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226217 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:29:02.236656 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226221 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:29:02.236656 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226224 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:29:02.236656 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226227 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:29:02.236656 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226230 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:29:02.236656 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226237 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:29:02.237102 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226239 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:29:02.237102 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226242 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:29:02.237102 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226246 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:29:02.237102 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226249 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:29:02.237102 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226252 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:29:02.237102 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226254 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:29:02.237102 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226257 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:29:02.237102 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226259 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:29:02.237102 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226262 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:29:02.237102 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226264 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:29:02.237102 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226267 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:29:02.237102 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226269 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:29:02.237102 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226272 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:29:02.237102 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226274 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:29:02.237102 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226277 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:29:02.237102 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226279 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:29:02.237102 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226282 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:29:02.237102 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226284 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:29:02.237102 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226286 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:29:02.237558 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226289 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:29:02.237558 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226292 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:29:02.237558 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226296 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:29:02.237558 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226315 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:29:02.237558 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226318 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:29:02.237558 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226321 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:29:02.237558 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226324 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:29:02.237558 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226326 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:29:02.237558 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226330 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:29:02.237558 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226333 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:29:02.237558 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226335 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:29:02.237558 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226338 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:29:02.237558 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226340 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:29:02.237558 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226344 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:29:02.237558 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226348 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:29:02.237558 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226353 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:29:02.237558 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226356 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:29:02.237558 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226359 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:29:02.237558 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226362 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:29:02.237558 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226365 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:29:02.238019 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226367 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:29:02.238019 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226370 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:29:02.238019 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226373 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:29:02.238019 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226375 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:29:02.238019 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226378 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:29:02.238019 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226381 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:29:02.238019 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226383 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:29:02.238019 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226386 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:29:02.238019 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226388 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:29:02.238019 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226391 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:29:02.238019 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226393 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:29:02.238019 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226396 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:29:02.238019 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226398 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:29:02.238019 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226401 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:29:02.238019 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226405 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:29:02.238019 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226408 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:29:02.238019 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226416 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:29:02.238019 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226419 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:29:02.238019 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226421 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:29:02.238019 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226424 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:29:02.238538 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226427 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:29:02.238538 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226429 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:29:02.238538 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226432 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:29:02.238538 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226434 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:29:02.238538 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226437 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:29:02.238538 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226439 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:29:02.238538 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226442 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:29:02.238538 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226446 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:29:02.238538 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226449 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:29:02.238538 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226451 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:29:02.238538 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226454 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:29:02.238538 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226457 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:29:02.238538 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226459 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:29:02.238538 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226462 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:29:02.238538 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226464 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:29:02.238538 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226467 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:29:02.238538 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226469 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:29:02.238538 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226472 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:29:02.238538 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226474 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:29:02.238990 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226477 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:29:02.238990 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226479 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:29:02.238990 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226483 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:29:02.238990 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226485 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:29:02.238990 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226488 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:29:02.238990 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226490 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:29:02.238990 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.226493 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:29:02.238990 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.234055 2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 17:29:02.238990 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.234077 2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 17:29:02.238990 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.234678 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 17:29:02.238990 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.234697 2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 17:29:02.238990 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.234951 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:29:02.238990 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.234964 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:29:02.238990 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.234968 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:29:02.238990 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.234971 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:29:02.238990 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.234975 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:29:02.238990 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.234977 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:29:02.238990 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.234980 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:29:02.238990 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.234983 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:29:02.238990 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.234986 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:29:02.239536 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.234989 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:29:02.239536 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.234991 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:29:02.239536 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.234994 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:29:02.239536 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.234997 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:29:02.239536 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235000 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:29:02.239536 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235003 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:29:02.239536 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235006 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:29:02.239536 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235008 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:29:02.239536 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235011 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:29:02.239536 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235013 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:29:02.239536 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235016 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:29:02.239536 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235020 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:29:02.239536 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235023 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:29:02.239536 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235025 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:29:02.239536 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235028 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:29:02.239536 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235031 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:29:02.239536 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235033 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:29:02.239536 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235036 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:29:02.239536 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235039 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:29:02.240002 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235041 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:29:02.240002 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235044 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:29:02.240002 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235047 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:29:02.240002 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235049 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:29:02.240002 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235052 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:29:02.240002 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235056 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:29:02.240002 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235058 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:29:02.240002 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235061 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:29:02.240002 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235064 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:29:02.240002 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235066 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:29:02.240002 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235069 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:29:02.240002 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235072 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:29:02.240002 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235074 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:29:02.240002 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235077 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:29:02.240002 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235079 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:29:02.240002 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235082 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:29:02.240002 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235084 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:29:02.240002 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235087 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:29:02.240002 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235090 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:29:02.240002 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235094 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:29:02.240490 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235097 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:29:02.240490 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235099 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:29:02.240490 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235102 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:29:02.240490 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235105 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:29:02.240490 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235108 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:29:02.240490 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235110 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:29:02.240490 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235113 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:29:02.240490 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235115 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:29:02.240490 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235120 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:29:02.240490 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235124 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:29:02.240490 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235128 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:29:02.240490 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235130 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:29:02.240490 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235133 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:29:02.240490 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235136 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:29:02.240490 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235138 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:29:02.240490 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235141 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:29:02.240490 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235144 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:29:02.240490 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235146 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:29:02.240490 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235149 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:29:02.240490 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235152 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:29:02.240948 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235154 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:29:02.240948 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235157 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:29:02.240948 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235160 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:29:02.240948 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235162 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:29:02.240948 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235165 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:29:02.240948 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235167 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:29:02.240948 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235170 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:29:02.240948 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235172 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:29:02.240948 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235175 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:29:02.240948 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235177 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:29:02.240948 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235180 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:29:02.240948 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235182 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:29:02.240948 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235185 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:29:02.240948 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235188 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:29:02.240948 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235191 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:29:02.240948 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235194 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:29:02.240948 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235196 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:29:02.240948 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235199 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:29:02.240948 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235201 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:29:02.241418 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235204 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:29:02.241418 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235207 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:29:02.241418 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235209 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:29:02.241418 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235212 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:29:02.241418 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235215 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:29:02.241418 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235217 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:29:02.241418 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235220 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:29:02.241418 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235222 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:29:02.241418 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235225 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:29:02.241418 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235228 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:29:02.241418 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235230 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:29:02.241418 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235233 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:29:02.241418 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235236 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:29:02.241418 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235238 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:29:02.241418 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235241 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:29:02.241418 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235243 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:29:02.241418 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235245 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:29:02.241418 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235248 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:29:02.241418 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235252 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:29:02.241872 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.235282 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:29:02.241872 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235440 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:29:02.241872 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235445 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:29:02.241872 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235449 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:29:02.241872 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235452 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:29:02.241872 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235455 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:29:02.241872 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235467 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:29:02.241872 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235471 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:29:02.241872 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235474 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:29:02.241872 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235478 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:29:02.241872 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235481 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:29:02.241872 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235484 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:29:02.241872 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235487 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:29:02.241872 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235490 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:29:02.241872 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235493 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:29:02.242225 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235495 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:29:02.242225 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235498 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:29:02.242225 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235500 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:29:02.242225 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235504 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:29:02.242225 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235508 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:29:02.242225 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235510 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:29:02.242225 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235513 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:29:02.242225 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235516 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:29:02.242225 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235519 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:29:02.242225 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235522 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:29:02.242225 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235524 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:29:02.242225 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235527 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:29:02.242225 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235529 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:29:02.242225 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235532 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:29:02.242225 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235535 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:29:02.242225 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235537 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:29:02.242225 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235540 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:29:02.242225 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235542 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:29:02.242225 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235545 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:29:02.242770 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235547 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:29:02.242770 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235550 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:29:02.242770 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235553 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:29:02.242770 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235555 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:29:02.242770 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235558 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:29:02.242770 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235566 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:29:02.242770 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235569 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:29:02.242770 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235572 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:29:02.242770 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235574 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:29:02.242770 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235577 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:29:02.242770 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235579 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:29:02.242770 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235582 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:29:02.242770 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235585 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:29:02.242770 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235587 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:29:02.242770 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235590 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:29:02.242770 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235592 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:29:02.242770 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235595 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:29:02.242770 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235597 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:29:02.242770 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235600 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:29:02.243221 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235602 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:29:02.243221 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235605 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:29:02.243221 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235607 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:29:02.243221 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235610 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:29:02.243221 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235613 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:29:02.243221 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235616 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:29:02.243221 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235618 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:29:02.243221 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235621 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:29:02.243221 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235623 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:29:02.243221 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235626 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:29:02.243221 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235628 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:29:02.243221 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235631 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:29:02.243221 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235634 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:29:02.243221 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235636 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:29:02.243221 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235641 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:29:02.243221 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235643 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:29:02.243221 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235646 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:29:02.243221 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235648 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:29:02.243221 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235651 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:29:02.243221 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235659 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:29:02.243776 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235661 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:29:02.243776 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235664 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:29:02.243776 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235666 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:29:02.243776 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235669 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:29:02.243776 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235671 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:29:02.243776 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235674 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:29:02.243776 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235676 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:29:02.243776 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235679 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:29:02.243776 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235681 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:29:02.243776 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235684 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:29:02.243776 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235686 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:29:02.243776 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235689 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:29:02.243776 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235691 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:29:02.243776 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235694 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:29:02.243776 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235698 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:29:02.243776 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235700 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:29:02.243776 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235703 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:29:02.243776 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235706 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:29:02.243776 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235708 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:29:02.243776 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235711 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:29:02.244245 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235714 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:29:02.244245 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235717 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:29:02.244245 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235719 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:29:02.244245 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235722 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:29:02.244245 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235724 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:29:02.244245 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235727 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:29:02.244245 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235730 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:29:02.244245 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235733 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:29:02.244245 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235736 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:29:02.244245 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235738 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:29:02.244245 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235741 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:29:02.244245 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235743 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:29:02.244245 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235747 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:29:02.244245 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235750 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:29:02.244245 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.235773 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:29:02.244245 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235804 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:29:02.244644 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235808 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:29:02.244644 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235811 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:29:02.244644 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235814 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:29:02.244644 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235817 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:29:02.244644 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235819 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:29:02.244644 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235822 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:29:02.244644 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235824 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:29:02.244644 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235827 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:29:02.244644 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235830 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:29:02.244644 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235832 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:29:02.244644 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235835 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:29:02.244644 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235838 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:29:02.244644 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235841 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:29:02.244644 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235844 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:29:02.244644 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235846 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:29:02.244644 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235849 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:29:02.244644 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235851 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:29:02.244644 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235854 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:29:02.244644 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235856 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:29:02.244644 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235859 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:29:02.245236 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235861 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:29:02.245236 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235864 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:29:02.245236 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235868 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:29:02.245236 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235870 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:29:02.245236 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235873 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:29:02.245236 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235875 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:29:02.245236 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235877 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:29:02.245236 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235880 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:29:02.245236 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235882 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:29:02.245236 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235886 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:29:02.245236 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235888 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:29:02.245236 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235890 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:29:02.245236 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235893 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:29:02.245236 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235895 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:29:02.245236 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235898 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:29:02.245236 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235901 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:29:02.245236 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235903 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:29:02.245236 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235906 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:29:02.245236 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235908 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:29:02.245236 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235911 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:29:02.245740 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235913 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:29:02.245740 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235916 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:29:02.245740 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235918 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:29:02.245740 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235921 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:29:02.245740 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235924 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:29:02.245740 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235926 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:29:02.245740 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235928 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:29:02.245740 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235931 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:29:02.245740 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235934 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:29:02.245740 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235936 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:29:02.245740 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235939 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:29:02.245740 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235941 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:29:02.245740 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235944 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:29:02.245740 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235946 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:29:02.245740 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235950 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:29:02.245740 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235953 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:29:02.245740 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235955 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:29:02.245740 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235958 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:29:02.245740 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235960 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:29:02.245740 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235963 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:29:02.246213 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235965 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:29:02.246213 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235968 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:29:02.246213 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235971 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:29:02.246213 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235974 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:29:02.246213 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235976 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:29:02.246213 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235979 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:29:02.246213 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235981 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:29:02.246213 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235984 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:29:02.246213 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235988 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:29:02.246213 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235992 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:29:02.246213 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235995 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:29:02.246213 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.235998 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:29:02.246213 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.236000 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:29:02.246213 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.236003 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:29:02.246213 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.236006 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:29:02.246213 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.236008 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:29:02.246213 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.236011 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:29:02.246213 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.236013 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:29:02.246213 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.236017 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:29:02.246702 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.236021 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:29:02.246702 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.236032 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:29:02.246702 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.236035 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:29:02.246702 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.236038 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:29:02.246702 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.236041 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:29:02.246702 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.236044 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:29:02.246702 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.236046 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:29:02.246702 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.236050 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:29:02.246702 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.236053 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:29:02.246702 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.236055 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:29:02.246702 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.236058 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:29:02.246702 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.236061 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:29:02.246702 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.236063 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:29:02.246702 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.236066 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:29:02.246702 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.236068 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:29:02.246702 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.236071 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:29:02.246702 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.236075 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:29:02.246702 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.236078 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:29:02.246702 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.236080 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:29:02.247266 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.236083 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:29:02.247266 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.236085 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:29:02.247266 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.236088 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:29:02.247266 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.236090 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:29:02.247266 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.236093 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:29:02.247266 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.236095 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:29:02.247266 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:02.236098 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:29:02.247266 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.236151 2580 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 17:29:02.247266 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.238885 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 17:29:02.247266 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.239685 2580 server.go:1014] "Starting client certificate rotation" Nov 02 17:29:02.247266 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.239780 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 17:29:02.247266 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.240401 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 17:29:02.247266 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.241488 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 17:29:02.247266 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.242542 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 17:29:02.253633 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.253616 2580 log.go:25] "Validated CRI v1 runtime API" Nov 02 17:29:02.256806 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.256791 2580 log.go:25] "Validated CRI v1 image API" Nov 02 17:29:02.257638 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.257616 2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 17:29:02.260404 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.260385 2580 fs.go:135] Filesystem UUIDs: map[20b21f6f-361d-4dee-8cd2-0fa25a362b67:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 86cc3286-62a7-4e7b-a5eb-d90d3e1f0913:/dev/nvme0n1p3] Nov 02 17:29:02.260481 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.260403 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 17:29:02.266177 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.266057 2580 manager.go:217] Machine: {Timestamp:2026-11-02 17:29:02.264021208 +0000 UTC m=+0.331566978 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3086838 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec258fbf820ecb9c6938a061d0e49ef5 SystemUUID:ec258fbf-820e-cb9c-6938-a061d0e49ef5 BootID:cc2062ec-e693-427d-8b0c-a65bb170ce11 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ec:fe:fa:b3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ec:fe:fa:b3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:86:c7:44:67:49 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 17:29:02.267066 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.267054 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 17:29:02.267174 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.267158 2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 17:29:02.270347 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.270320 2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 17:29:02.270467 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.270348 2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-26.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 17:29:02.270513 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.270470 2580 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 17:29:02.270513 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.270478 2580 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 17:29:02.270570 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.270514 2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 17:29:02.270570 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.270532 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 17:29:02.271374 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.271365 2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 17:29:02.274128 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.274117 2580 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 17:29:02.274241 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.274231 2580 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 17:29:02.274863 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.274852 2580 kubelet.go:493] "Attempting to sync node with API server" Nov 02 17:29:02.274893 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.274870 2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 17:29:02.274893 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.274881 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 17:29:02.274893 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.274890 2580 kubelet.go:405] "Adding apiserver pod source" Nov 02 17:29:02.275048 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.274900 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 17:29:02.275318 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.275283 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 17:29:02.278782 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.278759 2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 17:29:02.284279 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.284265 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 17:29:02.285093 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.285080 2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 17:29:02.285141 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.285111 2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 17:29:02.285280 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.285270 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 17:29:02.285333 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.285286 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 17:29:02.285333 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.285292 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 17:29:02.285333 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.285297 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 17:29:02.285333 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.285319 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 17:29:02.285333 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.285329 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 17:29:02.285898 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.285886 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 17:29:02.285928 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.285904 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 17:29:02.285928 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.285916 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 17:29:02.285928 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.285926 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 17:29:02.286004 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.285940 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 17:29:02.286004 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.285957 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 17:29:02.286601 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.286588 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 17:29:02.286639 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.286605 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 17:29:02.290411 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.290397 2580 server.go:1263] "Started kubelet" Nov 02 17:29:02.290531 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.290485 2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 17:29:02.290603 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.290553 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 17:29:02.290667 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.290627 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 17:29:02.291397 ip-10-0-142-26 systemd[1]: Started Kubernetes Kubelet. Nov 02 17:29:02.292135 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.292116 2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 17:29:02.293069 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.292941 2580 server.go:317] "Adding debug handlers to kubelet server" Nov 02 17:29:02.297115 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:02.297088 2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 17:29:02.297591 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.297572 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 17:29:02.298094 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.298061 2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 17:29:02.298289 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:02.298263 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-26.ec2.internal\" not found" Nov 02 17:29:02.298756 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.298736 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 17:29:02.298850 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.298746 2580 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 17:29:02.298850 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.298785 2580 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 17:29:02.298914 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.298883 2580 reconstruct.go:97] "Volume reconstruction finished" Nov 02 17:29:02.298914 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.298892 2580 reconciler.go:29] "Reconciler: start to sync state" Nov 02 17:29:02.299263 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.299247 2580 factory.go:55] Registering systemd factory Nov 02 17:29:02.299263 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.299264 2580 factory.go:223] Registration of the systemd container factory successfully Nov 02 17:29:02.299520 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.299503 2580 factory.go:153] Registering CRI-O factory Nov 02 17:29:02.299520 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.299522 2580 factory.go:223] Registration of the crio container factory successfully Nov 02 17:29:02.299668 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.299576 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 17:29:02.299668 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.299593 2580 factory.go:103] Registering Raw factory Nov 02 17:29:02.299668 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.299610 2580 manager.go:1199] Started watching for new ooms in manager Nov 02 17:29:02.301095 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.301080 2580 manager.go:319] Starting recovery of all containers Nov 02 17:29:02.311357 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.311335 2580 manager.go:324] Recovery completed Nov 02 17:29:02.315650 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.315633 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:29:02.317627 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.317601 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-26.ec2.internal" event="NodeHasSufficientMemory" Nov 02 17:29:02.317703 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.317641 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-26.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 17:29:02.317703 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.317652 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-26.ec2.internal" event="NodeHasSufficientPID" Nov 02 17:29:02.318205 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.318192 2580 cpu_manager.go:226] "Starting" policy="none" Nov 02 17:29:02.318323 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.318203 2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 17:29:02.318323 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.318233 2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 17:29:02.318323 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.318196 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:29:02.318990 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.318969 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-26.ec2.internal" event="NodeHasSufficientMemory" Nov 02 17:29:02.319183 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.318996 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-26.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 17:29:02.319183 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.319007 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-26.ec2.internal" event="NodeHasSufficientPID" Nov 02 17:29:02.320149 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.320135 2580 policy_none.go:50] "Start" Nov 02 17:29:02.320200 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.320152 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 17:29:02.320200 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.320173 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 17:29:02.321376 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.321362 2580 policy_none.go:44] "Start" Nov 02 17:29:02.325927 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.325906 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-26.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 17:29:02.325927 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:02.325914 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-26.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 17:29:02.328561 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:02.325909 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-26.ec2.internal.18e44a91f1f4d4f0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-26.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-26.ec2.internal,},FirstTimestamp:2026-11-02 17:29:02.290367728 +0000 UTC m=+0.357913500,LastTimestamp:2026-11-02 17:29:02.290367728 +0000 UTC m=+0.357913500,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-26.ec2.internal,}" Nov 02 17:29:02.335707 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.335682 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 17:29:02.335787 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.335737 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mtb8c" Nov 02 17:29:02.342488 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:02.342467 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 02 17:29:02.342740 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:02.342726 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 02 17:29:02.342983 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:02.342970 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-26.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 02 17:29:02.347765 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.347750 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mtb8c" Nov 02 17:29:02.348894 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.348867 2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 17:29:02.352359 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:02.352260 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-26.ec2.internal.18e44a91f394c4ed default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-26.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-26.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-26.ec2.internal,},FirstTimestamp:2026-11-02 17:29:02.317626605 +0000 UTC m=+0.385172375,LastTimestamp:2026-11-02 17:29:02.317626605 +0000 UTC m=+0.385172375,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-26.ec2.internal,}" Nov 02 17:29:02.352771 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.352756 2580 manager.go:341] "Starting Device Plugin manager" Nov 02 17:29:02.352853 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:02.352786 2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 17:29:02.352853 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.352823 2580 server.go:87] "Starting device plugin registration server" Nov 02 17:29:02.353090 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.353076 2580 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 17:29:02.353154 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.353089 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 17:29:02.353238 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.353219 2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 17:29:02.353337 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.353320 2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 17:29:02.353337 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.353333 2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 17:29:02.353857 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:02.353822 2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 17:29:02.353915 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:02.353886 2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-26.ec2.internal\" not found" Nov 02 17:29:02.369318 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:02.369228 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-26.ec2.internal.18e44a91f3950fe4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-26.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-26.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-26.ec2.internal,},FirstTimestamp:2026-11-02 17:29:02.317645796 +0000 UTC m=+0.385191567,LastTimestamp:2026-11-02 17:29:02.317645796 +0000 UTC m=+0.385191567,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-26.ec2.internal,}" Nov 02 17:29:02.403179 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:02.403092 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-26.ec2.internal.18e44a91f3953867 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-26.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-142-26.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-142-26.ec2.internal,},FirstTimestamp:2026-11-02 17:29:02.317656167 +0000 UTC m=+0.385201937,LastTimestamp:2026-11-02 17:29:02.317656167 +0000 UTC m=+0.385201937,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-26.ec2.internal,}" Nov 02 17:29:02.453213 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.453167 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:29:02.454337 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.454320 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-26.ec2.internal" event="NodeHasSufficientMemory" Nov 02 17:29:02.454441 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.454356 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-26.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 17:29:02.454441 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.454370 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-26.ec2.internal" event="NodeHasSufficientPID" Nov 02 17:29:02.454441 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.454401 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-26.ec2.internal" Nov 02 17:29:02.455712 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:02.455634 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-142-26.ec2.internal.18e44a91f394c4ed\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-26.ec2.internal.18e44a91f394c4ed default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-26.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-26.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-26.ec2.internal,},FirstTimestamp:2026-11-02 17:29:02.317626605 +0000 UTC m=+0.385172375,LastTimestamp:2026-11-02 17:29:02.318985519 +0000 UTC m=+0.386531297,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-26.ec2.internal,}" Nov 02 17:29:02.509289 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.509124 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 17:29:02.510706 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.510687 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 17:29:02.510795 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.510712 2580 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 17:29:02.510795 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.510730 2580 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 17:29:02.510795 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:02.510761 2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 17:29:02.516634 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.516613 2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-26.ec2.internal" Nov 02 17:29:02.516712 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:02.516640 2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-26.ec2.internal\": node \"ip-10-0-142-26.ec2.internal\" not found" Nov 02 17:29:02.527906 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.527887 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:29:02.583549 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:02.583522 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-26.ec2.internal\" not found" Nov 02 17:29:02.610977 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.610955 2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-26.ec2.internal"] Nov 02 17:29:02.611061 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.611036 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:29:02.612054 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.612039 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-26.ec2.internal" event="NodeHasSufficientMemory" Nov 02 17:29:02.612123 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.612066 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-26.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 17:29:02.612123 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.612076 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-26.ec2.internal" event="NodeHasSufficientPID" Nov 02 17:29:02.613758 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.613746 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:29:02.613915 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.613899 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal" Nov 02 17:29:02.613975 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.613936 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:29:02.616490 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.616454 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-26.ec2.internal" event="NodeHasSufficientMemory" Nov 02 17:29:02.616490 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.616464 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-26.ec2.internal" event="NodeHasSufficientMemory" Nov 02 17:29:02.616490 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.616484 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-26.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 17:29:02.616490 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.616488 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-26.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 17:29:02.616667 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.616496 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-26.ec2.internal" event="NodeHasSufficientPID" Nov 02 17:29:02.616667 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.616501 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-26.ec2.internal" event="NodeHasSufficientPID" Nov 02 17:29:02.617704 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.617687 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-26.ec2.internal" Nov 02 17:29:02.617753 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.617721 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:29:02.618493 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.618480 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-26.ec2.internal" event="NodeHasSufficientMemory" Nov 02 17:29:02.618581 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.618505 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-26.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 17:29:02.618581 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.618514 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-26.ec2.internal" event="NodeHasSufficientPID" Nov 02 17:29:02.634682 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:02.634661 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-26.ec2.internal\" not found" node="ip-10-0-142-26.ec2.internal" Nov 02 17:29:02.638361 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:02.638346 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-26.ec2.internal\" not found" node="ip-10-0-142-26.ec2.internal" Nov 02 17:29:02.684439 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:02.684419 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-26.ec2.internal\" not found" Nov 02 17:29:02.700672 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.700653 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/041bb4fbe95805a40957f9ee210f4b23-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal\" (UID: \"041bb4fbe95805a40957f9ee210f4b23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal" Nov 02 17:29:02.700740 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.700677 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/041bb4fbe95805a40957f9ee210f4b23-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal\" (UID: \"041bb4fbe95805a40957f9ee210f4b23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal" Nov 02 17:29:02.700740 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.700695 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1b001a1b5ad5b960c3a58390390fe0a6-config\") pod \"kube-apiserver-proxy-ip-10-0-142-26.ec2.internal\" (UID: \"1b001a1b5ad5b960c3a58390390fe0a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-26.ec2.internal" Nov 02 17:29:02.784537 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:02.784509 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-26.ec2.internal\" not found" Nov 02 17:29:02.800855 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.800832 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/041bb4fbe95805a40957f9ee210f4b23-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal\" (UID: \"041bb4fbe95805a40957f9ee210f4b23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal" Nov 02 17:29:02.800939 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.800859 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/041bb4fbe95805a40957f9ee210f4b23-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal\" (UID: \"041bb4fbe95805a40957f9ee210f4b23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal" Nov 02 17:29:02.800939 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.800875 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1b001a1b5ad5b960c3a58390390fe0a6-config\") pod \"kube-apiserver-proxy-ip-10-0-142-26.ec2.internal\" (UID: \"1b001a1b5ad5b960c3a58390390fe0a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-26.ec2.internal" Nov 02 17:29:02.800939 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.800923 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1b001a1b5ad5b960c3a58390390fe0a6-config\") pod \"kube-apiserver-proxy-ip-10-0-142-26.ec2.internal\" (UID: \"1b001a1b5ad5b960c3a58390390fe0a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-26.ec2.internal" Nov 02 17:29:02.801067 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.800950 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/041bb4fbe95805a40957f9ee210f4b23-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal\" (UID: \"041bb4fbe95805a40957f9ee210f4b23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal" Nov 02 17:29:02.801067 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.800951 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/041bb4fbe95805a40957f9ee210f4b23-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal\" (UID: \"041bb4fbe95805a40957f9ee210f4b23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal" Nov 02 17:29:02.884807 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:02.884785 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-26.ec2.internal\" not found" Nov 02 17:29:02.937856 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.937830 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal" Nov 02 17:29:02.941862 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:02.941498 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-26.ec2.internal" Nov 02 17:29:02.985516 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:02.985493 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-26.ec2.internal\" not found" Nov 02 17:29:03.086074 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:03.086010 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-26.ec2.internal\" not found" Nov 02 17:29:03.188890 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:03.188848 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-26.ec2.internal\" not found" Nov 02 17:29:03.241252 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:03.241226 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 17:29:03.241706 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:03.241381 2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 17:29:03.289650 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:03.289627 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-26.ec2.internal\" not found" Nov 02 17:29:03.316012 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:03.315992 2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:29:03.398827 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:03.398797 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal" Nov 02 17:29:03.425393 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:03.425346 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:29:03.426901 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:03.426887 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-26.ec2.internal" Nov 02 17:29:03.453208 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:03.453182 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod041bb4fbe95805a40957f9ee210f4b23.slice/crio-8d764223f8703f2c068280733e8df67e4b15cf89bec7e9f059b0baae9c2d2e91 WatchSource:0}: Error finding container 8d764223f8703f2c068280733e8df67e4b15cf89bec7e9f059b0baae9c2d2e91: Status 404 returned error can't find the container with id 8d764223f8703f2c068280733e8df67e4b15cf89bec7e9f059b0baae9c2d2e91 Nov 02 17:29:03.453596 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:03.453581 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b001a1b5ad5b960c3a58390390fe0a6.slice/crio-80dcc1078096451d0c8da3583979ee66c7a924f6d74409220f0c7ebee2ad3c46 WatchSource:0}: Error finding container 80dcc1078096451d0c8da3583979ee66c7a924f6d74409220f0c7ebee2ad3c46: Status 404 returned error can't find the container with id 80dcc1078096451d0c8da3583979ee66c7a924f6d74409220f0c7ebee2ad3c46 Nov 02 17:29:03.460499 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:03.460477 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:29:03.462451 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:03.462426 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 17:24:02 +0000 UTC" deadline="2028-03-28 11:19:15.084174961 +0000 UTC" Nov 02 17:29:03.462515 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:03.462451 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12281h50m11.621726981s" Nov 02 17:29:03.469859 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:03.469830 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:29:03.514590 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:03.514550 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal" event={"ID":"041bb4fbe95805a40957f9ee210f4b23","Type":"ContainerStarted","Data":"8d764223f8703f2c068280733e8df67e4b15cf89bec7e9f059b0baae9c2d2e91"} Nov 02 17:29:03.515637 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:03.515615 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-26.ec2.internal" event={"ID":"1b001a1b5ad5b960c3a58390390fe0a6","Type":"ContainerStarted","Data":"80dcc1078096451d0c8da3583979ee66c7a924f6d74409220f0c7ebee2ad3c46"} Nov 02 17:29:03.806242 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:03.806216 2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:29:03.912643 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:03.912612 2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:29:04.276665 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.276635 2580 apiserver.go:52] "Watching apiserver" Nov 02 17:29:04.299032 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.299003 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 17:29:04.302599 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.302576 2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 17:29:04.303033 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.303008 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-26.ec2.internal","openshift-cluster-node-tuning-operator/tuned-nsqnc","openshift-dns/node-resolver-22fc9","openshift-multus/network-metrics-daemon-fdtgx","openshift-network-diagnostics/network-check-target-hsdff","kube-system/global-pull-secret-syncer-l2q87","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal","openshift-network-operator/iptables-alerter-bscjw","kube-system/konnectivity-agent-rt5cp","openshift-multus/multus-7h4wg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg","openshift-image-registry/node-ca-bz42x","openshift-multus/multus-additional-cni-plugins-phn5m","openshift-ovn-kubernetes/ovnkube-node-8zq8j"] Nov 02 17:29:04.304829 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.304806 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.306221 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.306202 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:04.307690 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.307673 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-22fc9" Nov 02 17:29:04.308915 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.308894 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:04.309051 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:04.308983 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdtgx" podUID="5cb013c0-4887-488b-867f-841fb6b01ca5" Nov 02 17:29:04.310247 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.310227 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsdff" Nov 02 17:29:04.310366 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:04.310294 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsdff" podUID="d09569ff-f9b3-42d4-b163-fae46612bffc" Nov 02 17:29:04.311612 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.311593 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:04.311708 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:04.311678 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2q87" podUID="83c827e3-5338-49b5-97a1-56bdb5904500" Nov 02 17:29:04.313551 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.313528 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rt5cp" Nov 02 17:29:04.315544 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.315518 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.317017 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.316997 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.318466 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.318211 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bscjw" Nov 02 17:29:04.320761 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.320738 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qbqb7\"" Nov 02 17:29:04.321402 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.321359 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 17:29:04.321515 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.321496 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 17:29:04.322291 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.322266 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 17:29:04.322499 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.322482 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:04.328373 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.328322 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bz42x" Nov 02 17:29:04.330705 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.330682 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xdl26\"" Nov 02 17:29:04.350680 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.350657 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 17:29:04.350780 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.350703 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-blvvd" Nov 02 17:29:04.367105 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.367084 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-blvvd" Nov 02 17:29:04.400124 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.400100 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 17:29:04.410613 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.410581 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-kubelet\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.410747 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.410631 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-ovnkube-config\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.410747 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.410657 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-sysctl-conf\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.410747 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.410685 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b-host-slash\") pod \"iptables-alerter-bscjw\" (UID: \"fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b\") " pod="openshift-network-operator/iptables-alerter-bscjw" Nov 02 17:29:04.410747 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.410712 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-socket-dir\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:04.410747 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.410737 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-multus-conf-dir\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.411002 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.410764 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-sys\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.411002 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.410798 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fskz\" (UniqueName: \"kubernetes.io/projected/61135006-920f-4fb5-8ba5-c98d6beae6cf-kube-api-access-9fskz\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.411002 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.410827 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0aed3a0d-f6c0-4955-8d67-8b8b98bebe25-serviceca\") pod \"node-ca-bz42x\" (UID: \"0aed3a0d-f6c0-4955-8d67-8b8b98bebe25\") " pod="openshift-image-registry/node-ca-bz42x" Nov 02 17:29:04.411002 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.410866 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35ab9b40-d988-40ff-87c0-0cf822cfd863-tuning-conf-dir\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:04.411002 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.410920 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35ab9b40-d988-40ff-87c0-0cf822cfd863-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:04.411002 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.410972 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/83c827e3-5338-49b5-97a1-56bdb5904500-kubelet-config\") pod \"global-pull-secret-syncer-l2q87\" (UID: \"83c827e3-5338-49b5-97a1-56bdb5904500\") " pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:04.411257 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411019 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-cnibin\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.411257 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411049 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-multus-socket-dir-parent\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.411257 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411083 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv9jt\" (UniqueName: \"kubernetes.io/projected/d09569ff-f9b3-42d4-b163-fae46612bffc-kube-api-access-cv9jt\") pod \"network-check-target-hsdff\" (UID: \"d09569ff-f9b3-42d4-b163-fae46612bffc\") " pod="openshift-network-diagnostics/network-check-target-hsdff" Nov 02 17:29:04.411257 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411108 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-run-openvswitch\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.411257 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411133 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-log-socket\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.411257 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411158 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-registration-dir\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:04.411257 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411213 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c38cc772-7b2a-4a50-b230-e8b790647ae0-cni-binary-copy\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.411257 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411238 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/665ac753-596e-4d13-b8c1-7a1c7e791cf8-hosts-file\") pod \"node-resolver-22fc9\" (UID: \"665ac753-596e-4d13-b8c1-7a1c7e791cf8\") " pod="openshift-dns/node-resolver-22fc9" Nov 02 17:29:04.411627 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411263 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-run-netns\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.411627 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411280 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.411627 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411298 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-kubernetes\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.411627 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411319 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgkcf\" (UniqueName: \"kubernetes.io/projected/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-kube-api-access-zgkcf\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:04.411627 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411367 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-run-k8s-cni-cncf-io\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.411627 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411386 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-lib-modules\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.411627 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411452 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rcc6\" (UniqueName: \"kubernetes.io/projected/35ab9b40-d988-40ff-87c0-0cf822cfd863-kube-api-access-4rcc6\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:04.411627 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411473 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-var-lib-cni-bin\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.411627 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411488 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-systemd-units\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.411627 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411503 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61135006-920f-4fb5-8ba5-c98d6beae6cf-tmp\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.411627 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411517 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/665ac753-596e-4d13-b8c1-7a1c7e791cf8-tmp-dir\") pod \"node-resolver-22fc9\" (UID: \"665ac753-596e-4d13-b8c1-7a1c7e791cf8\") " pod="openshift-dns/node-resolver-22fc9" Nov 02 17:29:04.411627 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411548 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24tpd\" (UniqueName: \"kubernetes.io/projected/665ac753-596e-4d13-b8c1-7a1c7e791cf8-kube-api-access-24tpd\") pod \"node-resolver-22fc9\" (UID: \"665ac753-596e-4d13-b8c1-7a1c7e791cf8\") " pod="openshift-dns/node-resolver-22fc9" Nov 02 17:29:04.411627 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411580 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-slash\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.411627 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411596 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-hostroot\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.411627 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411623 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c38cc772-7b2a-4a50-b230-e8b790647ae0-multus-daemon-config\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.412265 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411643 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-etc-kubernetes\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.412265 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411661 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-run-ovn\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.412265 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411678 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-node-log\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.412265 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411693 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-cni-netd\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.412265 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411708 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-modprobe-d\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.412265 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411723 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-etc-selinux\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:04.412265 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411738 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-sys-fs\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:04.412265 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411811 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-run-netns\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.412265 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411829 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-ovn-node-metrics-cert\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.412265 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411850 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0aed3a0d-f6c0-4955-8d67-8b8b98bebe25-host\") pod \"node-ca-bz42x\" (UID: \"0aed3a0d-f6c0-4955-8d67-8b8b98bebe25\") " pod="openshift-image-registry/node-ca-bz42x" Nov 02 17:29:04.412265 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411892 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-device-dir\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:04.412265 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.411942 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-var-lib-kubelet\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.412265 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.412027 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-run-multus-certs\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.412265 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.412074 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-etc-openvswitch\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.412265 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.412110 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-cni-bin\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.412265 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.412137 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-sysctl-d\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.412265 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.412158 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-run\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.412974 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.412186 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35ab9b40-d988-40ff-87c0-0cf822cfd863-system-cni-dir\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:04.412974 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.412204 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b57h\" (UniqueName: \"kubernetes.io/projected/fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b-kube-api-access-8b57h\") pod \"iptables-alerter-bscjw\" (UID: \"fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b\") " pod="openshift-network-operator/iptables-alerter-bscjw" Nov 02 17:29:04.412974 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.412240 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-system-cni-dir\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.412974 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.412262 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83c827e3-5338-49b5-97a1-56bdb5904500-original-pull-secret\") pod \"global-pull-secret-syncer-l2q87\" (UID: \"83c827e3-5338-49b5-97a1-56bdb5904500\") " pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:04.412974 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.412291 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-systemd\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.412974 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.412311 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-var-lib-kubelet\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.412974 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.412360 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b-iptables-alerter-script\") pod \"iptables-alerter-bscjw\" (UID: \"fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b\") " pod="openshift-network-operator/iptables-alerter-bscjw" Nov 02 17:29:04.412974 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.412376 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-multus-cni-dir\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.412974 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.412400 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-run-systemd\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.412974 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.412457 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-ovnkube-script-lib\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.412974 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.412474 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-host\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.412974 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.412498 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-tuned\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.412974 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.412530 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1b5ee10f-87a1-4168-904e-ba233f304fe3-agent-certs\") pod \"konnectivity-agent-rt5cp\" (UID: \"1b5ee10f-87a1-4168-904e-ba233f304fe3\") " pod="kube-system/konnectivity-agent-rt5cp" Nov 02 17:29:04.412974 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.412550 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-kubelet-dir\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:04.412974 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.412570 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-sysconfig\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.412974 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.412595 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35ab9b40-d988-40ff-87c0-0cf822cfd863-cnibin\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:04.413726 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.412619 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35ab9b40-d988-40ff-87c0-0cf822cfd863-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:04.413726 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.412644 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-var-lib-cni-multus\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.413726 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.412665 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgsrn\" (UniqueName: \"kubernetes.io/projected/5cb013c0-4887-488b-867f-841fb6b01ca5-kube-api-access-dgsrn\") pod \"network-metrics-daemon-fdtgx\" (UID: \"5cb013c0-4887-488b-867f-841fb6b01ca5\") " pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:04.413726 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.412683 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-var-lib-openvswitch\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.413726 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.412699 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-run-ovn-kubernetes\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.413726 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.412732 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcgm2\" (UniqueName: \"kubernetes.io/projected/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-kube-api-access-jcgm2\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.413726 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.412748 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg9fq\" (UniqueName: \"kubernetes.io/projected/0aed3a0d-f6c0-4955-8d67-8b8b98bebe25-kube-api-access-cg9fq\") pod \"node-ca-bz42x\" (UID: \"0aed3a0d-f6c0-4955-8d67-8b8b98bebe25\") " pod="openshift-image-registry/node-ca-bz42x" Nov 02 17:29:04.413726 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.412841 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1b5ee10f-87a1-4168-904e-ba233f304fe3-konnectivity-ca\") pod \"konnectivity-agent-rt5cp\" (UID: \"1b5ee10f-87a1-4168-904e-ba233f304fe3\") " pod="kube-system/konnectivity-agent-rt5cp" Nov 02 17:29:04.413726 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.412862 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88l9x\" (UniqueName: \"kubernetes.io/projected/c38cc772-7b2a-4a50-b230-e8b790647ae0-kube-api-access-88l9x\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.413726 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.412883 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-env-overrides\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.413726 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.412899 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35ab9b40-d988-40ff-87c0-0cf822cfd863-cni-binary-copy\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:04.413726 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.412916 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/83c827e3-5338-49b5-97a1-56bdb5904500-dbus\") pod \"global-pull-secret-syncer-l2q87\" (UID: \"83c827e3-5338-49b5-97a1-56bdb5904500\") " pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:04.413726 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.412932 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cb013c0-4887-488b-867f-841fb6b01ca5-metrics-certs\") pod \"network-metrics-daemon-fdtgx\" (UID: \"5cb013c0-4887-488b-867f-841fb6b01ca5\") " pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:04.430013 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.429984 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 17:29:04.430165 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.430035 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:29:04.430276 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.430163 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 17:29:04.430276 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.430201 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 17:29:04.430276 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.430246 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 17:29:04.430276 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.430250 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 17:29:04.430620 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.430279 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 17:29:04.430620 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.430302 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 17:29:04.430620 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.430343 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 17:29:04.430620 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.430359 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 17:29:04.430620 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.430377 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 17:29:04.430620 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.430378 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 17:29:04.430620 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.430390 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:29:04.430620 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.430361 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 17:29:04.430620 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.430396 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 17:29:04.430620 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.430418 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 17:29:04.430620 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.430425 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 17:29:04.430620 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.430436 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 17:29:04.430620 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.430454 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 17:29:04.430620 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.430485 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 17:29:04.430620 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.430602 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 17:29:04.431366 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.431345 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 17:29:04.431595 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.431576 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 17:29:04.431694 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.431639 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jl8vw\"" Nov 02 17:29:04.431694 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.431645 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 17:29:04.431795 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.431773 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-mwsx7\"" Nov 02 17:29:04.431880 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.431837 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9lngf\"" Nov 02 17:29:04.431965 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.431948 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t2hrp\"" Nov 02 17:29:04.432040 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.431982 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bl7zd\"" Nov 02 17:29:04.432040 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.432032 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4b9fl\"" Nov 02 17:29:04.432107 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.432081 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vjnnw\"" Nov 02 17:29:04.510241 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.510208 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:29:04.513537 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.513493 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-system-cni-dir\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.513680 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.513538 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83c827e3-5338-49b5-97a1-56bdb5904500-original-pull-secret\") pod \"global-pull-secret-syncer-l2q87\" (UID: \"83c827e3-5338-49b5-97a1-56bdb5904500\") " pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:04.513680 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.513569 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-systemd\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.513680 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.513597 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-var-lib-kubelet\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.513680 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.513625 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b-iptables-alerter-script\") pod \"iptables-alerter-bscjw\" (UID: \"fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b\") " pod="openshift-network-operator/iptables-alerter-bscjw" Nov 02 17:29:04.513680 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.513637 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-system-cni-dir\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.513680 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.513649 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-multus-cni-dir\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.513991 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.513691 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-run-systemd\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.513991 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.513722 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-ovnkube-script-lib\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.513991 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.513748 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-host\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.513991 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.513772 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-tuned\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.513991 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.513799 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1b5ee10f-87a1-4168-904e-ba233f304fe3-agent-certs\") pod \"konnectivity-agent-rt5cp\" (UID: \"1b5ee10f-87a1-4168-904e-ba233f304fe3\") " pod="kube-system/konnectivity-agent-rt5cp" Nov 02 17:29:04.513991 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.513826 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-kubelet-dir\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:04.513991 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.513852 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-sysconfig\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.513991 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.513877 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35ab9b40-d988-40ff-87c0-0cf822cfd863-cnibin\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:04.513991 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.513884 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-multus-cni-dir\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.513991 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.513913 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35ab9b40-d988-40ff-87c0-0cf822cfd863-cnibin\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:04.513991 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.513928 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35ab9b40-d988-40ff-87c0-0cf822cfd863-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:04.513991 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:04.513936 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:29:04.514592 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.514012 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-systemd\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.514592 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:04.514021 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83c827e3-5338-49b5-97a1-56bdb5904500-original-pull-secret podName:83c827e3-5338-49b5-97a1-56bdb5904500 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:05.013997858 +0000 UTC m=+3.081543634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/83c827e3-5338-49b5-97a1-56bdb5904500-original-pull-secret") pod "global-pull-secret-syncer-l2q87" (UID: "83c827e3-5338-49b5-97a1-56bdb5904500") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:29:04.514592 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.514069 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-var-lib-kubelet\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.514592 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.514507 2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 17:29:04.514819 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.514706 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b-iptables-alerter-script\") pod \"iptables-alerter-bscjw\" (UID: \"fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b\") " pod="openshift-network-operator/iptables-alerter-bscjw" Nov 02 17:29:04.514819 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.514735 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-kubelet-dir\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:04.514819 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.514767 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-run-systemd\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.514819 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.514798 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-sysconfig\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.515697 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.515661 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-var-lib-cni-multus\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.515816 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.515712 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-host\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.516009 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.515991 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35ab9b40-d988-40ff-87c0-0cf822cfd863-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:04.516062 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.516044 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-var-lib-cni-multus\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.516134 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.516119 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgsrn\" (UniqueName: \"kubernetes.io/projected/5cb013c0-4887-488b-867f-841fb6b01ca5-kube-api-access-dgsrn\") pod \"network-metrics-daemon-fdtgx\" (UID: \"5cb013c0-4887-488b-867f-841fb6b01ca5\") " pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:04.516758 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.516725 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-ovnkube-script-lib\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.517152 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.517125 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-var-lib-openvswitch\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.517248 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.517178 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-run-ovn-kubernetes\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.517248 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.517213 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcgm2\" (UniqueName: \"kubernetes.io/projected/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-kube-api-access-jcgm2\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.517361 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.517245 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg9fq\" (UniqueName: \"kubernetes.io/projected/0aed3a0d-f6c0-4955-8d67-8b8b98bebe25-kube-api-access-cg9fq\") pod \"node-ca-bz42x\" (UID: \"0aed3a0d-f6c0-4955-8d67-8b8b98bebe25\") " pod="openshift-image-registry/node-ca-bz42x" Nov 02 17:29:04.517361 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.517277 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1b5ee10f-87a1-4168-904e-ba233f304fe3-konnectivity-ca\") pod \"konnectivity-agent-rt5cp\" (UID: \"1b5ee10f-87a1-4168-904e-ba233f304fe3\") " pod="kube-system/konnectivity-agent-rt5cp" Nov 02 17:29:04.517361 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.517302 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88l9x\" (UniqueName: \"kubernetes.io/projected/c38cc772-7b2a-4a50-b230-e8b790647ae0-kube-api-access-88l9x\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.517361 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.517347 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-env-overrides\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.517545 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.517377 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35ab9b40-d988-40ff-87c0-0cf822cfd863-cni-binary-copy\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:04.517545 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.517411 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/83c827e3-5338-49b5-97a1-56bdb5904500-dbus\") pod \"global-pull-secret-syncer-l2q87\" (UID: \"83c827e3-5338-49b5-97a1-56bdb5904500\") " pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:04.517545 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.517441 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cb013c0-4887-488b-867f-841fb6b01ca5-metrics-certs\") pod \"network-metrics-daemon-fdtgx\" (UID: \"5cb013c0-4887-488b-867f-841fb6b01ca5\") " pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:04.517545 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.517466 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-kubelet\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.517545 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.517497 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-ovnkube-config\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.517545 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.517526 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-sysctl-conf\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.517830 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.517558 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b-host-slash\") pod \"iptables-alerter-bscjw\" (UID: \"fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b\") " pod="openshift-network-operator/iptables-alerter-bscjw" Nov 02 17:29:04.517830 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.517585 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-socket-dir\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:04.517830 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.517616 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-multus-conf-dir\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.517830 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.517645 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-sys\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.517830 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.517685 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fskz\" (UniqueName: \"kubernetes.io/projected/61135006-920f-4fb5-8ba5-c98d6beae6cf-kube-api-access-9fskz\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.517830 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.517718 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0aed3a0d-f6c0-4955-8d67-8b8b98bebe25-serviceca\") pod \"node-ca-bz42x\" (UID: \"0aed3a0d-f6c0-4955-8d67-8b8b98bebe25\") " pod="openshift-image-registry/node-ca-bz42x" Nov 02 17:29:04.517830 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.517743 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35ab9b40-d988-40ff-87c0-0cf822cfd863-tuning-conf-dir\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:04.517830 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.517776 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35ab9b40-d988-40ff-87c0-0cf822cfd863-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:04.517830 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.517808 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/83c827e3-5338-49b5-97a1-56bdb5904500-kubelet-config\") pod \"global-pull-secret-syncer-l2q87\" (UID: \"83c827e3-5338-49b5-97a1-56bdb5904500\") " pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:04.518224 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.517847 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-cnibin\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.518224 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.517879 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-multus-socket-dir-parent\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.518224 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.517906 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv9jt\" (UniqueName: \"kubernetes.io/projected/d09569ff-f9b3-42d4-b163-fae46612bffc-kube-api-access-cv9jt\") pod \"network-check-target-hsdff\" (UID: \"d09569ff-f9b3-42d4-b163-fae46612bffc\") " pod="openshift-network-diagnostics/network-check-target-hsdff" Nov 02 17:29:04.518224 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.517939 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-run-openvswitch\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.518224 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.517969 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-log-socket\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.518224 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.517999 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-registration-dir\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:04.518224 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.518024 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c38cc772-7b2a-4a50-b230-e8b790647ae0-cni-binary-copy\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.518224 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.518057 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/665ac753-596e-4d13-b8c1-7a1c7e791cf8-hosts-file\") pod \"node-resolver-22fc9\" (UID: \"665ac753-596e-4d13-b8c1-7a1c7e791cf8\") " pod="openshift-dns/node-resolver-22fc9" Nov 02 17:29:04.518224 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.518087 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-run-netns\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.518224 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.518119 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.518224 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.518151 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-kubernetes\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.518224 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.518177 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgkcf\" (UniqueName: \"kubernetes.io/projected/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-kube-api-access-zgkcf\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:04.518224 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.518207 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-run-k8s-cni-cncf-io\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.518861 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.518237 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-lib-modules\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.518861 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.518269 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rcc6\" (UniqueName: \"kubernetes.io/projected/35ab9b40-d988-40ff-87c0-0cf822cfd863-kube-api-access-4rcc6\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:04.518861 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.518299 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-var-lib-cni-bin\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.518861 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.518329 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-systemd-units\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.518861 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.518359 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61135006-920f-4fb5-8ba5-c98d6beae6cf-tmp\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.518861 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.518402 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/665ac753-596e-4d13-b8c1-7a1c7e791cf8-tmp-dir\") pod \"node-resolver-22fc9\" (UID: \"665ac753-596e-4d13-b8c1-7a1c7e791cf8\") " pod="openshift-dns/node-resolver-22fc9" Nov 02 17:29:04.518861 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.518438 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24tpd\" (UniqueName: \"kubernetes.io/projected/665ac753-596e-4d13-b8c1-7a1c7e791cf8-kube-api-access-24tpd\") pod \"node-resolver-22fc9\" (UID: \"665ac753-596e-4d13-b8c1-7a1c7e791cf8\") " pod="openshift-dns/node-resolver-22fc9" Nov 02 17:29:04.518861 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.518466 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-slash\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.518861 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.518504 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-hostroot\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.518861 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.518535 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c38cc772-7b2a-4a50-b230-e8b790647ae0-multus-daemon-config\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.518861 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.518566 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-etc-kubernetes\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.518861 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.518596 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-run-ovn\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.518861 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.518621 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-node-log\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.518861 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.518652 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-cni-netd\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.518861 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.518685 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-modprobe-d\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.518861 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.518718 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-etc-selinux\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:04.518861 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.518752 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-sys-fs\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:04.518861 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.518765 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-var-lib-openvswitch\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.519697 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.518782 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-run-netns\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.519697 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.518870 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-log-socket\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.519697 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.518915 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-registration-dir\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:04.519697 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.518932 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-tuned\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.519697 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.518951 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-run-ovn-kubernetes\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.520319 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.520281 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-slash\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.520319 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.520291 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/665ac753-596e-4d13-b8c1-7a1c7e791cf8-tmp-dir\") pod \"node-resolver-22fc9\" (UID: \"665ac753-596e-4d13-b8c1-7a1c7e791cf8\") " pod="openshift-dns/node-resolver-22fc9" Nov 02 17:29:04.520456 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.520363 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b-host-slash\") pod \"iptables-alerter-bscjw\" (UID: \"fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b\") " pod="openshift-network-operator/iptables-alerter-bscjw" Nov 02 17:29:04.520456 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.520412 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-sys-fs\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:04.520559 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.520455 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-hostroot\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.520559 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.520517 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-socket-dir\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:04.520656 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.520567 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-multus-conf-dir\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.520656 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.520623 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-sys\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.523285 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.521084 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-etc-selinux\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:04.523285 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:04.521093 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:29:04.523285 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.521123 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-ovn-node-metrics-cert\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.523285 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.521130 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-lib-modules\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.523285 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:04.521161 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cb013c0-4887-488b-867f-841fb6b01ca5-metrics-certs podName:5cb013c0-4887-488b-867f-841fb6b01ca5 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:05.021135055 +0000 UTC m=+3.088680816 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5cb013c0-4887-488b-867f-841fb6b01ca5-metrics-certs") pod "network-metrics-daemon-fdtgx" (UID: "5cb013c0-4887-488b-867f-841fb6b01ca5") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:29:04.523285 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.521188 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-run-netns\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.523285 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.521200 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0aed3a0d-f6c0-4955-8d67-8b8b98bebe25-host\") pod \"node-ca-bz42x\" (UID: \"0aed3a0d-f6c0-4955-8d67-8b8b98bebe25\") " pod="openshift-image-registry/node-ca-bz42x" Nov 02 17:29:04.523285 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.521241 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-device-dir\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:04.523285 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.521251 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/83c827e3-5338-49b5-97a1-56bdb5904500-kubelet-config\") pod \"global-pull-secret-syncer-l2q87\" (UID: \"83c827e3-5338-49b5-97a1-56bdb5904500\") " pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:04.523285 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.521277 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-var-lib-kubelet\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.523285 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.521281 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35ab9b40-d988-40ff-87c0-0cf822cfd863-tuning-conf-dir\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:04.523285 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.521313 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-run-multus-certs\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.523285 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.521320 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-run-k8s-cni-cncf-io\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.523285 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.521348 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-etc-openvswitch\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.523285 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.521358 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-var-lib-cni-bin\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.523285 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.521398 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-cni-bin\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.523285 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.521468 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-sysctl-d\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.524521 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.521502 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-run\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.524521 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.521552 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35ab9b40-d988-40ff-87c0-0cf822cfd863-system-cni-dir\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:04.524521 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.521588 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b57h\" (UniqueName: \"kubernetes.io/projected/fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b-kube-api-access-8b57h\") pod \"iptables-alerter-bscjw\" (UID: \"fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b\") " pod="openshift-network-operator/iptables-alerter-bscjw" Nov 02 17:29:04.524521 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.521782 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0aed3a0d-f6c0-4955-8d67-8b8b98bebe25-serviceca\") pod \"node-ca-bz42x\" (UID: \"0aed3a0d-f6c0-4955-8d67-8b8b98bebe25\") " pod="openshift-image-registry/node-ca-bz42x" Nov 02 17:29:04.524521 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.521892 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35ab9b40-d988-40ff-87c0-0cf822cfd863-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:04.524521 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.521909 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c38cc772-7b2a-4a50-b230-e8b790647ae0-multus-daemon-config\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.524521 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.521972 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-etc-kubernetes\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.524521 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.521997 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-run-openvswitch\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.524521 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.522027 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-run-ovn\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.524521 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.522123 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/665ac753-596e-4d13-b8c1-7a1c7e791cf8-hosts-file\") pod \"node-resolver-22fc9\" (UID: \"665ac753-596e-4d13-b8c1-7a1c7e791cf8\") " pod="openshift-dns/node-resolver-22fc9" Nov 02 17:29:04.524521 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.522178 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-run-netns\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.524521 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.522244 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-kubelet\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.524521 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.522299 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1b5ee10f-87a1-4168-904e-ba233f304fe3-agent-certs\") pod \"konnectivity-agent-rt5cp\" (UID: \"1b5ee10f-87a1-4168-904e-ba233f304fe3\") " pod="kube-system/konnectivity-agent-rt5cp" Nov 02 17:29:04.524521 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.522396 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.524521 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.522441 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-node-log\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.524521 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.522454 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-env-overrides\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.524521 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.522508 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1b5ee10f-87a1-4168-904e-ba233f304fe3-konnectivity-ca\") pod \"konnectivity-agent-rt5cp\" (UID: \"1b5ee10f-87a1-4168-904e-ba233f304fe3\") " pod="kube-system/konnectivity-agent-rt5cp" Nov 02 17:29:04.524521 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.522513 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-systemd-units\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.525319 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.522572 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-multus-socket-dir-parent\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.525319 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.522676 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-modprobe-d\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.525319 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.522688 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c38cc772-7b2a-4a50-b230-e8b790647ae0-cni-binary-copy\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.525319 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.522737 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-device-dir\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:04.525319 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.522848 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-kubernetes\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.525319 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.522775 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/83c827e3-5338-49b5-97a1-56bdb5904500-dbus\") pod \"global-pull-secret-syncer-l2q87\" (UID: \"83c827e3-5338-49b5-97a1-56bdb5904500\") " pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:04.525319 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.522940 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35ab9b40-d988-40ff-87c0-0cf822cfd863-cni-binary-copy\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:04.525319 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.522991 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61135006-920f-4fb5-8ba5-c98d6beae6cf-tmp\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.525319 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.523037 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-cni-netd\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.525319 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.523056 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-run-multus-certs\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.525319 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.523100 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-etc-openvswitch\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.525319 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.523130 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-sysctl-conf\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.525319 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.523152 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-run\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.525319 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.523205 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-var-lib-kubelet\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.525319 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.523225 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-cni-bin\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.525319 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.523236 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-sysctl-d\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.525319 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.523237 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-cnibin\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.525319 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.523220 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0aed3a0d-f6c0-4955-8d67-8b8b98bebe25-host\") pod \"node-ca-bz42x\" (UID: \"0aed3a0d-f6c0-4955-8d67-8b8b98bebe25\") " pod="openshift-image-registry/node-ca-bz42x" Nov 02 17:29:04.526262 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.523378 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35ab9b40-d988-40ff-87c0-0cf822cfd863-system-cni-dir\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:04.526262 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.524224 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-ovnkube-config\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.526943 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.526877 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-ovn-node-metrics-cert\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.572538 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.572474 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rcc6\" (UniqueName: \"kubernetes.io/projected/35ab9b40-d988-40ff-87c0-0cf822cfd863-kube-api-access-4rcc6\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:04.575253 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.575196 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcgm2\" (UniqueName: \"kubernetes.io/projected/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-kube-api-access-jcgm2\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.576309 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:04.575610 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:29:04.576309 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:04.575639 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:29:04.576309 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:04.575654 2580 projected.go:196] Error preparing data for projected volume kube-api-access-cv9jt for pod openshift-network-diagnostics/network-check-target-hsdff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:29:04.576309 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:04.575718 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d09569ff-f9b3-42d4-b163-fae46612bffc-kube-api-access-cv9jt podName:d09569ff-f9b3-42d4-b163-fae46612bffc nodeName:}" failed. No retries permitted until 2026-11-02 17:29:05.07569898 +0000 UTC m=+3.143244759 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cv9jt" (UniqueName: "kubernetes.io/projected/d09569ff-f9b3-42d4-b163-fae46612bffc-kube-api-access-cv9jt") pod "network-check-target-hsdff" (UID: "d09569ff-f9b3-42d4-b163-fae46612bffc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:29:04.577291 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.577264 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b57h\" (UniqueName: \"kubernetes.io/projected/fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b-kube-api-access-8b57h\") pod \"iptables-alerter-bscjw\" (UID: \"fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b\") " pod="openshift-network-operator/iptables-alerter-bscjw" Nov 02 17:29:04.577291 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.577275 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24tpd\" (UniqueName: \"kubernetes.io/projected/665ac753-596e-4d13-b8c1-7a1c7e791cf8-kube-api-access-24tpd\") pod \"node-resolver-22fc9\" (UID: \"665ac753-596e-4d13-b8c1-7a1c7e791cf8\") " pod="openshift-dns/node-resolver-22fc9" Nov 02 17:29:04.577863 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.577837 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88l9x\" (UniqueName: \"kubernetes.io/projected/c38cc772-7b2a-4a50-b230-e8b790647ae0-kube-api-access-88l9x\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.579193 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.579147 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgsrn\" (UniqueName: \"kubernetes.io/projected/5cb013c0-4887-488b-867f-841fb6b01ca5-kube-api-access-dgsrn\") pod \"network-metrics-daemon-fdtgx\" (UID: \"5cb013c0-4887-488b-867f-841fb6b01ca5\") " pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:04.580106 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.580080 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgkcf\" (UniqueName: \"kubernetes.io/projected/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-kube-api-access-zgkcf\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:04.580268 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.580233 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg9fq\" (UniqueName: \"kubernetes.io/projected/0aed3a0d-f6c0-4955-8d67-8b8b98bebe25-kube-api-access-cg9fq\") pod \"node-ca-bz42x\" (UID: \"0aed3a0d-f6c0-4955-8d67-8b8b98bebe25\") " pod="openshift-image-registry/node-ca-bz42x" Nov 02 17:29:04.581119 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.581095 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fskz\" (UniqueName: \"kubernetes.io/projected/61135006-920f-4fb5-8ba5-c98d6beae6cf-kube-api-access-9fskz\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.621958 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.621932 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7h4wg" Nov 02 17:29:04.632470 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:04.632441 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc38cc772_7b2a_4a50_b230_e8b790647ae0.slice/crio-e68c4c0e2265daed54e886e09441c2e9d3230250f828fd57699f5ab28fbcd936 WatchSource:0}: Error finding container e68c4c0e2265daed54e886e09441c2e9d3230250f828fd57699f5ab28fbcd936: Status 404 returned error can't find the container with id e68c4c0e2265daed54e886e09441c2e9d3230250f828fd57699f5ab28fbcd936 Nov 02 17:29:04.636912 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.636888 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:04.640160 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.639985 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-22fc9" Nov 02 17:29:04.644110 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.644087 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rt5cp" Nov 02 17:29:04.647014 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:04.646898 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35ab9b40_d988_40ff_87c0_0cf822cfd863.slice/crio-1857c4a0aa8bf1c3921d4ad577b26ade06fc0d3309705f5aebabf1cce95a1b1b WatchSource:0}: Error finding container 1857c4a0aa8bf1c3921d4ad577b26ade06fc0d3309705f5aebabf1cce95a1b1b: Status 404 returned error can't find the container with id 1857c4a0aa8bf1c3921d4ad577b26ade06fc0d3309705f5aebabf1cce95a1b1b Nov 02 17:29:04.652747 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:04.652710 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod665ac753_596e_4d13_b8c1_7a1c7e791cf8.slice/crio-e77524a03d85053d8c4fd96128de6a0e0add551fcae0300e84c5f2811506a1e3 WatchSource:0}: Error finding container e77524a03d85053d8c4fd96128de6a0e0add551fcae0300e84c5f2811506a1e3: Status 404 returned error can't find the container with id e77524a03d85053d8c4fd96128de6a0e0add551fcae0300e84c5f2811506a1e3 Nov 02 17:29:04.654180 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:04.654155 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b5ee10f_87a1_4168_904e_ba233f304fe3.slice/crio-a3499a029d1384764477256caca4907bf36720f5824334057f9a4b48f9732ecb WatchSource:0}: Error finding container a3499a029d1384764477256caca4907bf36720f5824334057f9a4b48f9732ecb: Status 404 returned error can't find the container with id a3499a029d1384764477256caca4907bf36720f5824334057f9a4b48f9732ecb Nov 02 17:29:04.665727 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.665705 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:04.665949 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.665926 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:04.670893 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.670742 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bscjw" Nov 02 17:29:04.677136 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:04.677108 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e5d6c0_3d7a_4c21_b44a_f0fd09f3b2a9.slice/crio-4037f20116599db676be2e3508ddaab64598b45fdcd9bfabeef17b59157040bc WatchSource:0}: Error finding container 4037f20116599db676be2e3508ddaab64598b45fdcd9bfabeef17b59157040bc: Status 404 returned error can't find the container with id 4037f20116599db676be2e3508ddaab64598b45fdcd9bfabeef17b59157040bc Nov 02 17:29:04.683059 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:04.683037 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61135006_920f_4fb5_8ba5_c98d6beae6cf.slice/crio-60133211eccbc8afe7c51ed446e31ccb5829fa3fa8a0363bb54024b392e47cec WatchSource:0}: Error finding container 60133211eccbc8afe7c51ed446e31ccb5829fa3fa8a0363bb54024b392e47cec: Status 404 returned error can't find the container with id 60133211eccbc8afe7c51ed446e31ccb5829fa3fa8a0363bb54024b392e47cec Nov 02 17:29:04.683680 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.683649 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:04.684468 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:04.684446 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe1a6d6b_5d70_4e68_ad95_1a7b7943b76b.slice/crio-d59d6ed36bf74eafed1a8ed10162ada70e1d312a26ed4f76ddb9c16631588825 WatchSource:0}: Error finding container d59d6ed36bf74eafed1a8ed10162ada70e1d312a26ed4f76ddb9c16631588825: Status 404 returned error can't find the container with id d59d6ed36bf74eafed1a8ed10162ada70e1d312a26ed4f76ddb9c16631588825 Nov 02 17:29:04.687495 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:04.686839 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bz42x" Nov 02 17:29:04.696086 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:04.696060 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b0e99b5_4469_44ac_b241_a6edb43bc8ec.slice/crio-32cb396d6ecfbbd76e588b0385a3c086a521faf4491e3bb2b2130e7ac232ea85 WatchSource:0}: Error finding container 32cb396d6ecfbbd76e588b0385a3c086a521faf4491e3bb2b2130e7ac232ea85: Status 404 returned error can't find the container with id 32cb396d6ecfbbd76e588b0385a3c086a521faf4491e3bb2b2130e7ac232ea85 Nov 02 17:29:04.697989 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:04.697801 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aed3a0d_f6c0_4955_8d67_8b8b98bebe25.slice/crio-aa47ee906c3a9ec04b42805201a94244a95b55047a0d8a219e86cb829b53fae6 WatchSource:0}: Error finding container aa47ee906c3a9ec04b42805201a94244a95b55047a0d8a219e86cb829b53fae6: Status 404 returned error can't find the container with id aa47ee906c3a9ec04b42805201a94244a95b55047a0d8a219e86cb829b53fae6 Nov 02 17:29:05.024337 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:05.024305 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83c827e3-5338-49b5-97a1-56bdb5904500-original-pull-secret\") pod \"global-pull-secret-syncer-l2q87\" (UID: \"83c827e3-5338-49b5-97a1-56bdb5904500\") " pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:05.024475 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:05.024448 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cb013c0-4887-488b-867f-841fb6b01ca5-metrics-certs\") pod \"network-metrics-daemon-fdtgx\" (UID: \"5cb013c0-4887-488b-867f-841fb6b01ca5\") " pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:05.024646 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:05.024624 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:29:05.024730 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:05.024703 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cb013c0-4887-488b-867f-841fb6b01ca5-metrics-certs podName:5cb013c0-4887-488b-867f-841fb6b01ca5 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:06.024683034 +0000 UTC m=+4.092228800 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5cb013c0-4887-488b-867f-841fb6b01ca5-metrics-certs") pod "network-metrics-daemon-fdtgx" (UID: "5cb013c0-4887-488b-867f-841fb6b01ca5") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:29:05.024815 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:05.024802 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:29:05.024869 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:05.024841 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83c827e3-5338-49b5-97a1-56bdb5904500-original-pull-secret podName:83c827e3-5338-49b5-97a1-56bdb5904500 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:06.024830591 +0000 UTC m=+4.092376364 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/83c827e3-5338-49b5-97a1-56bdb5904500-original-pull-secret") pod "global-pull-secret-syncer-l2q87" (UID: "83c827e3-5338-49b5-97a1-56bdb5904500") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:29:05.125866 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:05.125832 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv9jt\" (UniqueName: \"kubernetes.io/projected/d09569ff-f9b3-42d4-b163-fae46612bffc-kube-api-access-cv9jt\") pod \"network-check-target-hsdff\" (UID: \"d09569ff-f9b3-42d4-b163-fae46612bffc\") " pod="openshift-network-diagnostics/network-check-target-hsdff" Nov 02 17:29:05.126035 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:05.126019 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:29:05.126100 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:05.126040 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:29:05.126100 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:05.126053 2580 projected.go:196] Error preparing data for projected volume kube-api-access-cv9jt for pod openshift-network-diagnostics/network-check-target-hsdff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:29:05.126203 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:05.126106 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d09569ff-f9b3-42d4-b163-fae46612bffc-kube-api-access-cv9jt podName:d09569ff-f9b3-42d4-b163-fae46612bffc nodeName:}" failed. No retries permitted until 2026-11-02 17:29:06.12608801 +0000 UTC m=+4.193633777 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cv9jt" (UniqueName: "kubernetes.io/projected/d09569ff-f9b3-42d4-b163-fae46612bffc-kube-api-access-cv9jt") pod "network-check-target-hsdff" (UID: "d09569ff-f9b3-42d4-b163-fae46612bffc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:29:05.369687 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:05.369282 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 17:24:04 +0000 UTC" deadline="2028-06-02 09:59:01.160359682 +0000 UTC" Nov 02 17:29:05.369687 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:05.369597 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13864h29m55.790768726s" Nov 02 17:29:05.511401 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:05.511361 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:05.511401 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:05.511497 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2q87" podUID="83c827e3-5338-49b5-97a1-56bdb5904500" Nov 02 17:29:05.537704 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:05.537669 2580 generic.go:356] "Generic (PLEG): container finished" podID="041bb4fbe95805a40957f9ee210f4b23" containerID="801975d5c0d815c18e05c96a17e575f860556d85eea18898988f9a5c4e034aaf" exitCode=0 Nov 02 17:29:05.537864 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:05.537765 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal" event={"ID":"041bb4fbe95805a40957f9ee210f4b23","Type":"ContainerDied","Data":"801975d5c0d815c18e05c96a17e575f860556d85eea18898988f9a5c4e034aaf"} Nov 02 17:29:05.547938 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:05.547416 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-26.ec2.internal" event={"ID":"1b001a1b5ad5b960c3a58390390fe0a6","Type":"ContainerStarted","Data":"01c0c1a10661f609509a068380ec11c6e00c6deacbd11a9076c80fbd86d78691"} Nov 02 17:29:05.553877 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:05.553848 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bz42x" event={"ID":"0aed3a0d-f6c0-4955-8d67-8b8b98bebe25","Type":"ContainerStarted","Data":"aa47ee906c3a9ec04b42805201a94244a95b55047a0d8a219e86cb829b53fae6"} Nov 02 17:29:05.558837 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:05.558809 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" event={"ID":"61135006-920f-4fb5-8ba5-c98d6beae6cf","Type":"ContainerStarted","Data":"60133211eccbc8afe7c51ed446e31ccb5829fa3fa8a0363bb54024b392e47cec"} Nov 02 17:29:05.562843 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:05.561416 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" event={"ID":"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9","Type":"ContainerStarted","Data":"4037f20116599db676be2e3508ddaab64598b45fdcd9bfabeef17b59157040bc"} Nov 02 17:29:05.564481 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:05.564448 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rt5cp" event={"ID":"1b5ee10f-87a1-4168-904e-ba233f304fe3","Type":"ContainerStarted","Data":"a3499a029d1384764477256caca4907bf36720f5824334057f9a4b48f9732ecb"} Nov 02 17:29:05.572431 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:05.572407 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-22fc9" event={"ID":"665ac753-596e-4d13-b8c1-7a1c7e791cf8","Type":"ContainerStarted","Data":"e77524a03d85053d8c4fd96128de6a0e0add551fcae0300e84c5f2811506a1e3"} Nov 02 17:29:05.592646 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:05.592615 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" event={"ID":"0b0e99b5-4469-44ac-b241-a6edb43bc8ec","Type":"ContainerStarted","Data":"32cb396d6ecfbbd76e588b0385a3c086a521faf4491e3bb2b2130e7ac232ea85"} Nov 02 17:29:05.596550 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:05.596524 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bscjw" event={"ID":"fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b","Type":"ContainerStarted","Data":"d59d6ed36bf74eafed1a8ed10162ada70e1d312a26ed4f76ddb9c16631588825"} Nov 02 17:29:05.608466 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:05.608438 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phn5m" event={"ID":"35ab9b40-d988-40ff-87c0-0cf822cfd863","Type":"ContainerStarted","Data":"1857c4a0aa8bf1c3921d4ad577b26ade06fc0d3309705f5aebabf1cce95a1b1b"} Nov 02 17:29:05.615783 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:05.615754 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7h4wg" event={"ID":"c38cc772-7b2a-4a50-b230-e8b790647ae0","Type":"ContainerStarted","Data":"e68c4c0e2265daed54e886e09441c2e9d3230250f828fd57699f5ab28fbcd936"} Nov 02 17:29:05.631095 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:05.631006 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-26.ec2.internal" podStartSLOduration=2.630988145 podStartE2EDuration="2.630988145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:29:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:05.630984963 +0000 UTC m=+3.698530748" watchObservedRunningTime="2026-11-02 17:29:05.630988145 +0000 UTC m=+3.698533932" Nov 02 17:29:06.044714 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:06.044673 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cb013c0-4887-488b-867f-841fb6b01ca5-metrics-certs\") pod \"network-metrics-daemon-fdtgx\" (UID: \"5cb013c0-4887-488b-867f-841fb6b01ca5\") " pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:06.044902 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:06.044778 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83c827e3-5338-49b5-97a1-56bdb5904500-original-pull-secret\") pod \"global-pull-secret-syncer-l2q87\" (UID: \"83c827e3-5338-49b5-97a1-56bdb5904500\") " pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:06.045048 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:06.044946 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:29:06.045048 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:06.045014 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83c827e3-5338-49b5-97a1-56bdb5904500-original-pull-secret podName:83c827e3-5338-49b5-97a1-56bdb5904500 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:08.044994588 +0000 UTC m=+6.112540349 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/83c827e3-5338-49b5-97a1-56bdb5904500-original-pull-secret") pod "global-pull-secret-syncer-l2q87" (UID: "83c827e3-5338-49b5-97a1-56bdb5904500") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:29:06.045475 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:06.045454 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:29:06.045571 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:06.045516 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cb013c0-4887-488b-867f-841fb6b01ca5-metrics-certs podName:5cb013c0-4887-488b-867f-841fb6b01ca5 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:08.045498459 +0000 UTC m=+6.113044226 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5cb013c0-4887-488b-867f-841fb6b01ca5-metrics-certs") pod "network-metrics-daemon-fdtgx" (UID: "5cb013c0-4887-488b-867f-841fb6b01ca5") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:29:06.145524 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:06.145483 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv9jt\" (UniqueName: \"kubernetes.io/projected/d09569ff-f9b3-42d4-b163-fae46612bffc-kube-api-access-cv9jt\") pod \"network-check-target-hsdff\" (UID: \"d09569ff-f9b3-42d4-b163-fae46612bffc\") " pod="openshift-network-diagnostics/network-check-target-hsdff" Nov 02 17:29:06.145744 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:06.145725 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:29:06.145837 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:06.145755 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:29:06.145837 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:06.145769 2580 projected.go:196] Error preparing data for projected volume kube-api-access-cv9jt for pod openshift-network-diagnostics/network-check-target-hsdff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:29:06.145837 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:06.145832 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d09569ff-f9b3-42d4-b163-fae46612bffc-kube-api-access-cv9jt podName:d09569ff-f9b3-42d4-b163-fae46612bffc nodeName:}" failed. No retries permitted until 2026-11-02 17:29:08.145814906 +0000 UTC m=+6.213360671 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cv9jt" (UniqueName: "kubernetes.io/projected/d09569ff-f9b3-42d4-b163-fae46612bffc-kube-api-access-cv9jt") pod "network-check-target-hsdff" (UID: "d09569ff-f9b3-42d4-b163-fae46612bffc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:29:06.370445 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:06.370331 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 17:24:04 +0000 UTC" deadline="2028-06-09 17:43:50.278994398 +0000 UTC" Nov 02 17:29:06.370445 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:06.370379 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14040h14m43.908620527s" Nov 02 17:29:06.512252 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:06.511574 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsdff" Nov 02 17:29:06.512252 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:06.511724 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsdff" podUID="d09569ff-f9b3-42d4-b163-fae46612bffc" Nov 02 17:29:06.512252 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:06.511572 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:06.512252 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:06.512197 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdtgx" podUID="5cb013c0-4887-488b-867f-841fb6b01ca5" Nov 02 17:29:06.635056 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:06.634971 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal" event={"ID":"041bb4fbe95805a40957f9ee210f4b23","Type":"ContainerStarted","Data":"4023237f41cd5189da5a1e6681bb6cc6aadb6aa8418266ba87d27403f367bf10"} Nov 02 17:29:06.675387 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:06.675281 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal" podStartSLOduration=3.675261404 podStartE2EDuration="3.675261404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:29:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:06.674561207 +0000 UTC m=+4.742107000" watchObservedRunningTime="2026-11-02 17:29:06.675261404 +0000 UTC m=+4.742807189" Nov 02 17:29:07.511666 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:07.511633 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:07.512176 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:07.511768 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2q87" podUID="83c827e3-5338-49b5-97a1-56bdb5904500" Nov 02 17:29:07.645979 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:07.645737 2580 generic.go:356] "Generic (PLEG): container finished" podID="35ab9b40-d988-40ff-87c0-0cf822cfd863" containerID="668e946404e576494c689fbf72fbf622aade945eaf6fd64c6a40fe8b9cd20bdb" exitCode=0 Nov 02 17:29:07.645979 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:07.645862 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phn5m" event={"ID":"35ab9b40-d988-40ff-87c0-0cf822cfd863","Type":"ContainerDied","Data":"668e946404e576494c689fbf72fbf622aade945eaf6fd64c6a40fe8b9cd20bdb"} Nov 02 17:29:08.063683 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:08.063642 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83c827e3-5338-49b5-97a1-56bdb5904500-original-pull-secret\") pod \"global-pull-secret-syncer-l2q87\" (UID: \"83c827e3-5338-49b5-97a1-56bdb5904500\") " pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:08.063856 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:08.063709 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cb013c0-4887-488b-867f-841fb6b01ca5-metrics-certs\") pod \"network-metrics-daemon-fdtgx\" (UID: \"5cb013c0-4887-488b-867f-841fb6b01ca5\") " pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:08.063856 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:08.063831 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:29:08.063974 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:08.063912 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cb013c0-4887-488b-867f-841fb6b01ca5-metrics-certs podName:5cb013c0-4887-488b-867f-841fb6b01ca5 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:12.063892997 +0000 UTC m=+10.131438782 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5cb013c0-4887-488b-867f-841fb6b01ca5-metrics-certs") pod "network-metrics-daemon-fdtgx" (UID: "5cb013c0-4887-488b-867f-841fb6b01ca5") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:29:08.064383 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:08.064337 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:29:08.064493 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:08.064393 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83c827e3-5338-49b5-97a1-56bdb5904500-original-pull-secret podName:83c827e3-5338-49b5-97a1-56bdb5904500 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:12.06437799 +0000 UTC m=+10.131923753 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/83c827e3-5338-49b5-97a1-56bdb5904500-original-pull-secret") pod "global-pull-secret-syncer-l2q87" (UID: "83c827e3-5338-49b5-97a1-56bdb5904500") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:29:08.165553 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:08.164930 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv9jt\" (UniqueName: \"kubernetes.io/projected/d09569ff-f9b3-42d4-b163-fae46612bffc-kube-api-access-cv9jt\") pod \"network-check-target-hsdff\" (UID: \"d09569ff-f9b3-42d4-b163-fae46612bffc\") " pod="openshift-network-diagnostics/network-check-target-hsdff" Nov 02 17:29:08.165553 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:08.165140 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:29:08.165553 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:08.165162 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:29:08.165553 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:08.165173 2580 projected.go:196] Error preparing data for projected volume kube-api-access-cv9jt for pod openshift-network-diagnostics/network-check-target-hsdff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:29:08.165553 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:08.165224 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d09569ff-f9b3-42d4-b163-fae46612bffc-kube-api-access-cv9jt podName:d09569ff-f9b3-42d4-b163-fae46612bffc nodeName:}" failed. No retries permitted until 2026-11-02 17:29:12.16521112 +0000 UTC m=+10.232756901 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cv9jt" (UniqueName: "kubernetes.io/projected/d09569ff-f9b3-42d4-b163-fae46612bffc-kube-api-access-cv9jt") pod "network-check-target-hsdff" (UID: "d09569ff-f9b3-42d4-b163-fae46612bffc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:29:08.512976 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:08.512447 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:08.512976 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:08.512592 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdtgx" podUID="5cb013c0-4887-488b-867f-841fb6b01ca5" Nov 02 17:29:08.513762 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:08.513618 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsdff" Nov 02 17:29:08.513762 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:08.513726 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsdff" podUID="d09569ff-f9b3-42d4-b163-fae46612bffc" Nov 02 17:29:08.661169 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:08.661072 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rt5cp" event={"ID":"1b5ee10f-87a1-4168-904e-ba233f304fe3","Type":"ContainerStarted","Data":"b26bb2a9ba46686d498f89fff48760f4aba16002cefc108aadc06ad3afacb57f"} Nov 02 17:29:08.711683 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:08.711593 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rt5cp" podStartSLOduration=3.197354702 podStartE2EDuration="6.711571344s" totalImagesPullingTime="3.514216642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:08.711047446 +0000 UTC m=+6.778593229" watchObservedRunningTime="2026-11-02 17:29:08.711571344 +0000 UTC m=+6.779117126" Nov 02 17:29:09.511798 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:09.511293 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:09.511798 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:09.511453 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2q87" podUID="83c827e3-5338-49b5-97a1-56bdb5904500" Nov 02 17:29:10.514628 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:10.513847 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:10.514628 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:10.513999 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdtgx" podUID="5cb013c0-4887-488b-867f-841fb6b01ca5" Nov 02 17:29:10.514628 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:10.514410 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsdff" Nov 02 17:29:10.514628 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:10.514508 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsdff" podUID="d09569ff-f9b3-42d4-b163-fae46612bffc" Nov 02 17:29:10.668538 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:10.668458 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" event={"ID":"0b0e99b5-4469-44ac-b241-a6edb43bc8ec","Type":"ContainerStarted","Data":"bde661b7b4071b402a075e7dfac6cae0686a51b97782007527b5abf5a57a56df"} Nov 02 17:29:10.672814 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:10.672782 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bz42x" event={"ID":"0aed3a0d-f6c0-4955-8d67-8b8b98bebe25","Type":"ContainerStarted","Data":"40ccfe00e92ff6f72d39184ad6b38058822781c59276d0036cf2df0665c7671b"} Nov 02 17:29:11.512068 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:11.511063 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:11.512068 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:11.511195 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2q87" podUID="83c827e3-5338-49b5-97a1-56bdb5904500" Nov 02 17:29:11.802043 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:11.802015 2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 17:29:12.099953 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:12.099868 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83c827e3-5338-49b5-97a1-56bdb5904500-original-pull-secret\") pod \"global-pull-secret-syncer-l2q87\" (UID: \"83c827e3-5338-49b5-97a1-56bdb5904500\") " pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:12.099953 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:12.099929 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cb013c0-4887-488b-867f-841fb6b01ca5-metrics-certs\") pod \"network-metrics-daemon-fdtgx\" (UID: \"5cb013c0-4887-488b-867f-841fb6b01ca5\") " pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:12.100160 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:12.100025 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:29:12.100160 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:12.100052 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:29:12.100160 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:12.100100 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83c827e3-5338-49b5-97a1-56bdb5904500-original-pull-secret podName:83c827e3-5338-49b5-97a1-56bdb5904500 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:20.100080331 +0000 UTC m=+18.167626089 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/83c827e3-5338-49b5-97a1-56bdb5904500-original-pull-secret") pod "global-pull-secret-syncer-l2q87" (UID: "83c827e3-5338-49b5-97a1-56bdb5904500") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:29:12.100160 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:12.100119 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cb013c0-4887-488b-867f-841fb6b01ca5-metrics-certs podName:5cb013c0-4887-488b-867f-841fb6b01ca5 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:20.10011003 +0000 UTC m=+18.167655789 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5cb013c0-4887-488b-867f-841fb6b01ca5-metrics-certs") pod "network-metrics-daemon-fdtgx" (UID: "5cb013c0-4887-488b-867f-841fb6b01ca5") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:29:12.201161 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:12.201129 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv9jt\" (UniqueName: \"kubernetes.io/projected/d09569ff-f9b3-42d4-b163-fae46612bffc-kube-api-access-cv9jt\") pod \"network-check-target-hsdff\" (UID: \"d09569ff-f9b3-42d4-b163-fae46612bffc\") " pod="openshift-network-diagnostics/network-check-target-hsdff" Nov 02 17:29:12.201342 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:12.201321 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:29:12.201400 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:12.201352 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:29:12.201400 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:12.201365 2580 projected.go:196] Error preparing data for projected volume kube-api-access-cv9jt for pod openshift-network-diagnostics/network-check-target-hsdff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:29:12.201463 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:12.201423 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d09569ff-f9b3-42d4-b163-fae46612bffc-kube-api-access-cv9jt podName:d09569ff-f9b3-42d4-b163-fae46612bffc nodeName:}" failed. No retries permitted until 2026-11-02 17:29:20.201403959 +0000 UTC m=+18.268949719 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cv9jt" (UniqueName: "kubernetes.io/projected/d09569ff-f9b3-42d4-b163-fae46612bffc-kube-api-access-cv9jt") pod "network-check-target-hsdff" (UID: "d09569ff-f9b3-42d4-b163-fae46612bffc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:29:12.359657 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:12.358922 2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T17:29:11.802040776Z","UUID":"66999e26-e994-4d81-8906-f98c5739b58f","Handler":null,"Name":"","Endpoint":""} Nov 02 17:29:12.363137 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:12.362629 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 17:29:12.363137 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:12.362657 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 17:29:12.512628 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:12.512599 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:12.512783 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:12.512687 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdtgx" podUID="5cb013c0-4887-488b-867f-841fb6b01ca5" Nov 02 17:29:12.513098 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:12.513076 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsdff" Nov 02 17:29:12.513204 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:12.513182 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsdff" podUID="d09569ff-f9b3-42d4-b163-fae46612bffc" Nov 02 17:29:12.683178 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:12.683023 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" event={"ID":"0b0e99b5-4469-44ac-b241-a6edb43bc8ec","Type":"ContainerStarted","Data":"57f7ece59ea2e1de71f0afa9f62ea061bbef9729d749d8d43090c93d8b57d1cf"} Nov 02 17:29:12.687019 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:12.686674 2580 generic.go:356] "Generic (PLEG): container finished" podID="35ab9b40-d988-40ff-87c0-0cf822cfd863" containerID="5e07449ead7a871d88f150c122d63ee2b61fac4ef98ed9aa7e0e71b2846412ee" exitCode=0 Nov 02 17:29:12.687019 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:12.686721 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phn5m" event={"ID":"35ab9b40-d988-40ff-87c0-0cf822cfd863","Type":"ContainerDied","Data":"5e07449ead7a871d88f150c122d63ee2b61fac4ef98ed9aa7e0e71b2846412ee"} Nov 02 17:29:12.747697 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:12.747211 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bz42x" podStartSLOduration=5.636082011 podStartE2EDuration="10.747198402s" totalImagesPullingTime="5.111116391s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:10.711565046 +0000 UTC m=+8.779110827" watchObservedRunningTime="2026-11-02 17:29:12.747198402 +0000 UTC m=+10.814744181" Nov 02 17:29:13.272360 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:13.272144 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rt5cp" Nov 02 17:29:13.273506 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:13.273482 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rt5cp" Nov 02 17:29:13.511205 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:13.511162 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:13.511412 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:13.511300 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2q87" podUID="83c827e3-5338-49b5-97a1-56bdb5904500" Nov 02 17:29:13.691380 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:13.691288 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-22fc9" event={"ID":"665ac753-596e-4d13-b8c1-7a1c7e791cf8","Type":"ContainerStarted","Data":"48ada4d88a5432214a38fedc2a12a6f9868843feb561cf103b2bdb50fd1404aa"} Nov 02 17:29:13.694009 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:13.693981 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" event={"ID":"0b0e99b5-4469-44ac-b241-a6edb43bc8ec","Type":"ContainerStarted","Data":"a75f66f13761202efb1fbb418eea9ce953f24f300b266f798f7473254cfcf717"} Nov 02 17:29:13.697102 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:13.697076 2580 generic.go:356] "Generic (PLEG): container finished" podID="35ab9b40-d988-40ff-87c0-0cf822cfd863" containerID="b01b88595570d99a443b64de4cc005a7d1aac11f485e46b813656c301f57bf7d" exitCode=0 Nov 02 17:29:13.697206 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:13.697101 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phn5m" event={"ID":"35ab9b40-d988-40ff-87c0-0cf822cfd863","Type":"ContainerDied","Data":"b01b88595570d99a443b64de4cc005a7d1aac11f485e46b813656c301f57bf7d"} Nov 02 17:29:13.701027 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:13.700650 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7h4wg" event={"ID":"c38cc772-7b2a-4a50-b230-e8b790647ae0","Type":"ContainerStarted","Data":"083e9e74f3b6658726eec0ae3f710b519ff0586e1c0109e66f12571aa889a857"} Nov 02 17:29:13.701027 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:13.700688 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rt5cp" Nov 02 17:29:13.701146 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:13.701090 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rt5cp" Nov 02 17:29:13.727821 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:13.727778 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-22fc9" podStartSLOduration=3.627588839 podStartE2EDuration="11.727765171s" totalImagesPullingTime="8.100176332s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:13.727026222 +0000 UTC m=+11.794572003" watchObservedRunningTime="2026-11-02 17:29:13.727765171 +0000 UTC m=+11.795310952" Nov 02 17:29:13.917227 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:13.917154 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-7h4wg" podStartSLOduration=3.9199880929999997 podStartE2EDuration="11.917134908s" totalImagesPullingTime="7.997146815s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:13.845802058 +0000 UTC m=+11.913347839" watchObservedRunningTime="2026-11-02 17:29:13.917134908 +0000 UTC m=+11.984680689" Nov 02 17:29:14.514863 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:14.512711 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsdff" Nov 02 17:29:14.514863 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:14.512851 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsdff" podUID="d09569ff-f9b3-42d4-b163-fae46612bffc" Nov 02 17:29:14.514863 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:14.512899 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:14.514863 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:14.513030 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdtgx" podUID="5cb013c0-4887-488b-867f-841fb6b01ca5" Nov 02 17:29:14.703487 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:14.703412 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bscjw" event={"ID":"fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b","Type":"ContainerStarted","Data":"b4761431147ce331f3c27b68d67734cb903032315641670f7457cc24ed42012d"} Nov 02 17:29:14.706430 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:14.706405 2580 generic.go:356] "Generic (PLEG): container finished" podID="35ab9b40-d988-40ff-87c0-0cf822cfd863" containerID="0f36144c7e352ed92c970501fe7ef5d9618fde7de624c3c965a1a0ff7da335e4" exitCode=0 Nov 02 17:29:14.706537 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:14.706441 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phn5m" event={"ID":"35ab9b40-d988-40ff-87c0-0cf822cfd863","Type":"ContainerDied","Data":"0f36144c7e352ed92c970501fe7ef5d9618fde7de624c3c965a1a0ff7da335e4"} Nov 02 17:29:14.739707 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:14.739661 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bscjw" podStartSLOduration=4.777420241 podStartE2EDuration="12.739645142s" totalImagesPullingTime="7.962224901s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:14.739498887 +0000 UTC m=+12.807044666" watchObservedRunningTime="2026-11-02 17:29:14.739645142 +0000 UTC m=+12.807190923" Nov 02 17:29:14.739899 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:14.739872 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" podStartSLOduration=4.884329317 podStartE2EDuration="12.739862646s" totalImagesPullingTime="7.855533329s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:29:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:13.972693398 +0000 UTC m=+12.040239179" watchObservedRunningTime="2026-11-02 17:29:14.739862646 +0000 UTC m=+12.807408428" Nov 02 17:29:15.511290 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:15.511253 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:15.511486 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:15.511408 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2q87" podUID="83c827e3-5338-49b5-97a1-56bdb5904500" Nov 02 17:29:15.710926 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:15.710891 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" event={"ID":"61135006-920f-4fb5-8ba5-c98d6beae6cf","Type":"ContainerStarted","Data":"551c322bae85a2b0a383466119ec9e1f17224f73f3f77f7ac269f3e523a1ddd5"} Nov 02 17:29:15.760914 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:15.760853 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" podStartSLOduration=3.516747586 podStartE2EDuration="13.760833039s" totalImagesPullingTime="10.244085453s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:15.760758323 +0000 UTC m=+13.828304102" watchObservedRunningTime="2026-11-02 17:29:15.760833039 +0000 UTC m=+13.828378820" Nov 02 17:29:16.511625 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:16.511594 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsdff" Nov 02 17:29:16.511734 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:16.511639 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:16.511800 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:16.511740 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsdff" podUID="d09569ff-f9b3-42d4-b163-fae46612bffc" Nov 02 17:29:16.511855 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:16.511830 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdtgx" podUID="5cb013c0-4887-488b-867f-841fb6b01ca5" Nov 02 17:29:16.716906 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:16.716826 2580 generic.go:356] "Generic (PLEG): container finished" podID="35ab9b40-d988-40ff-87c0-0cf822cfd863" containerID="031a156231396530cc0ef872f168e0f630f8fd31be92843f443460d9e32bc106" exitCode=0 Nov 02 17:29:16.717479 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:16.716915 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phn5m" event={"ID":"35ab9b40-d988-40ff-87c0-0cf822cfd863","Type":"ContainerDied","Data":"031a156231396530cc0ef872f168e0f630f8fd31be92843f443460d9e32bc106"} Nov 02 17:29:17.511766 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:17.511561 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:17.511924 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:17.511865 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2q87" podUID="83c827e3-5338-49b5-97a1-56bdb5904500" Nov 02 17:29:17.723070 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:17.723037 2580 generic.go:356] "Generic (PLEG): container finished" podID="35ab9b40-d988-40ff-87c0-0cf822cfd863" containerID="a784eba432f8f5c6606aebe621b584d33399b2e2dab6dc0754ea0ca1566d4b57" exitCode=0 Nov 02 17:29:17.723491 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:17.723079 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phn5m" event={"ID":"35ab9b40-d988-40ff-87c0-0cf822cfd863","Type":"ContainerDied","Data":"a784eba432f8f5c6606aebe621b584d33399b2e2dab6dc0754ea0ca1566d4b57"} Nov 02 17:29:18.511703 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:18.511671 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsdff" Nov 02 17:29:18.511869 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:18.511718 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:18.511869 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:18.511808 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsdff" podUID="d09569ff-f9b3-42d4-b163-fae46612bffc" Nov 02 17:29:18.511989 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:18.511917 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdtgx" podUID="5cb013c0-4887-488b-867f-841fb6b01ca5" Nov 02 17:29:18.728162 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:18.728131 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phn5m" event={"ID":"35ab9b40-d988-40ff-87c0-0cf822cfd863","Type":"ContainerStarted","Data":"fb40ab12d4cbadda53a3d6ae3aa5c003be3ba7d6928e9e615b5a8b861427c6a0"} Nov 02 17:29:19.511642 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:19.511619 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:19.511773 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:19.511721 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2q87" podUID="83c827e3-5338-49b5-97a1-56bdb5904500" Nov 02 17:29:19.732691 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:19.732653 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" event={"ID":"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9","Type":"ContainerStarted","Data":"468040106c4de3dbf0f9a37e9efa6d290ad57123c83d42407f1e82e5875eb737"} Nov 02 17:29:19.733117 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:19.732731 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" event={"ID":"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9","Type":"ContainerStarted","Data":"cd46b4a51403024e069ce4a21de8851dcaccb4ffbdae6647a03dfbf6d8cdb376"} Nov 02 17:29:19.733117 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:19.732762 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" event={"ID":"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9","Type":"ContainerStarted","Data":"a374151c8f3e9bc2f5cb21626cfedb32b4aaf722c194a407e73a756612376433"} Nov 02 17:29:19.733117 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:19.732797 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" event={"ID":"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9","Type":"ContainerStarted","Data":"87511c8c90d4e4eddc7df6010f34b8d9b80d6998601612f731c3e2ff69b8b590"} Nov 02 17:29:19.733117 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:19.732833 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" event={"ID":"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9","Type":"ContainerStarted","Data":"ab56ace015b7949fbff420598d312eabc5b5da2648e4b663fe52dc57050936a1"} Nov 02 17:29:19.733117 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:19.732867 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" event={"ID":"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9","Type":"ContainerStarted","Data":"906046a2f386a4adcff659d09434a515eb4c7bc23f03a0ce637d0af1980c9c67"} Nov 02 17:29:20.166070 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:20.166034 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83c827e3-5338-49b5-97a1-56bdb5904500-original-pull-secret\") pod \"global-pull-secret-syncer-l2q87\" (UID: \"83c827e3-5338-49b5-97a1-56bdb5904500\") " pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:20.166070 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:20.166075 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cb013c0-4887-488b-867f-841fb6b01ca5-metrics-certs\") pod \"network-metrics-daemon-fdtgx\" (UID: \"5cb013c0-4887-488b-867f-841fb6b01ca5\") " pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:20.166235 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:20.166180 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:29:20.166281 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:20.166240 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83c827e3-5338-49b5-97a1-56bdb5904500-original-pull-secret podName:83c827e3-5338-49b5-97a1-56bdb5904500 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:36.166225422 +0000 UTC m=+34.233771221 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/83c827e3-5338-49b5-97a1-56bdb5904500-original-pull-secret") pod "global-pull-secret-syncer-l2q87" (UID: "83c827e3-5338-49b5-97a1-56bdb5904500") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:29:20.166281 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:20.166249 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:29:20.166366 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:20.166288 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cb013c0-4887-488b-867f-841fb6b01ca5-metrics-certs podName:5cb013c0-4887-488b-867f-841fb6b01ca5 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:36.166275595 +0000 UTC m=+34.233821352 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5cb013c0-4887-488b-867f-841fb6b01ca5-metrics-certs") pod "network-metrics-daemon-fdtgx" (UID: "5cb013c0-4887-488b-867f-841fb6b01ca5") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:29:20.266524 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:20.266481 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv9jt\" (UniqueName: \"kubernetes.io/projected/d09569ff-f9b3-42d4-b163-fae46612bffc-kube-api-access-cv9jt\") pod \"network-check-target-hsdff\" (UID: \"d09569ff-f9b3-42d4-b163-fae46612bffc\") " pod="openshift-network-diagnostics/network-check-target-hsdff" Nov 02 17:29:20.266661 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:20.266632 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:29:20.266661 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:20.266647 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:29:20.266661 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:20.266657 2580 projected.go:196] Error preparing data for projected volume kube-api-access-cv9jt for pod openshift-network-diagnostics/network-check-target-hsdff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:29:20.266774 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:20.266707 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d09569ff-f9b3-42d4-b163-fae46612bffc-kube-api-access-cv9jt podName:d09569ff-f9b3-42d4-b163-fae46612bffc nodeName:}" failed. No retries permitted until 2026-11-02 17:29:36.26669168 +0000 UTC m=+34.334237451 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cv9jt" (UniqueName: "kubernetes.io/projected/d09569ff-f9b3-42d4-b163-fae46612bffc-kube-api-access-cv9jt") pod "network-check-target-hsdff" (UID: "d09569ff-f9b3-42d4-b163-fae46612bffc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:29:20.510984 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:20.510907 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:20.511117 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:20.511019 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdtgx" podUID="5cb013c0-4887-488b-867f-841fb6b01ca5" Nov 02 17:29:20.511117 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:20.511076 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsdff" Nov 02 17:29:20.511206 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:20.511172 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsdff" podUID="d09569ff-f9b3-42d4-b163-fae46612bffc" Nov 02 17:29:21.511607 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:21.511575 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:21.512056 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:21.511678 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2q87" podUID="83c827e3-5338-49b5-97a1-56bdb5904500" Nov 02 17:29:21.742221 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:21.742126 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" event={"ID":"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9","Type":"ContainerStarted","Data":"946bd9f6bce0e52203e9a2b6352ea01461bf75c818bdfe284e8b6d8bce2211cf"} Nov 02 17:29:22.512206 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:22.512176 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:22.512589 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:22.512324 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdtgx" podUID="5cb013c0-4887-488b-867f-841fb6b01ca5" Nov 02 17:29:22.512589 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:22.512371 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsdff" Nov 02 17:29:22.512589 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:22.512489 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsdff" podUID="d09569ff-f9b3-42d4-b163-fae46612bffc" Nov 02 17:29:23.511082 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:23.511057 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:23.511179 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:23.511145 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2q87" podUID="83c827e3-5338-49b5-97a1-56bdb5904500" Nov 02 17:29:23.749960 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:23.749689 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" event={"ID":"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9","Type":"ContainerStarted","Data":"8055875354fd7cc9e501e8ba00e0273a585125692d4987d2bec7fc9efb989871"} Nov 02 17:29:23.749960 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:23.749924 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:23.750638 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:23.749967 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:23.764784 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:23.764762 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:23.839760 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:23.839713 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-phn5m" podStartSLOduration=11.886681517 podStartE2EDuration="21.839701298s" totalImagesPullingTime="9.953019781s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:29:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:18.792060058 +0000 UTC m=+16.859605849" watchObservedRunningTime="2026-11-02 17:29:23.839701298 +0000 UTC m=+21.907247077" Nov 02 17:29:23.839891 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:23.839814 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" podStartSLOduration=7.521929377 podStartE2EDuration="21.839809867s" totalImagesPullingTime="14.31788049s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:23.829644246 +0000 UTC m=+21.897190026" watchObservedRunningTime="2026-11-02 17:29:23.839809867 +0000 UTC m=+21.907355647" Nov 02 17:29:24.511116 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:24.511084 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:24.511251 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:24.511226 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdtgx" podUID="5cb013c0-4887-488b-867f-841fb6b01ca5" Nov 02 17:29:24.511320 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:24.511272 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsdff" Nov 02 17:29:24.511391 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:24.511374 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsdff" podUID="d09569ff-f9b3-42d4-b163-fae46612bffc" Nov 02 17:29:24.752506 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:24.752478 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:24.767492 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:24.767433 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:25.511963 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:25.511936 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:25.512121 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:25.512067 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2q87" podUID="83c827e3-5338-49b5-97a1-56bdb5904500" Nov 02 17:29:25.593361 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:25.593278 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l2q87"] Nov 02 17:29:25.596204 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:25.596182 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hsdff"] Nov 02 17:29:25.596336 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:25.596275 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsdff" Nov 02 17:29:25.596385 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:25.596368 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsdff" podUID="d09569ff-f9b3-42d4-b163-fae46612bffc" Nov 02 17:29:25.600205 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:25.600181 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fdtgx"] Nov 02 17:29:25.600300 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:25.600288 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:25.600408 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:25.600389 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdtgx" podUID="5cb013c0-4887-488b-867f-841fb6b01ca5" Nov 02 17:29:25.753604 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:25.753569 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:25.754088 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:25.753671 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2q87" podUID="83c827e3-5338-49b5-97a1-56bdb5904500" Nov 02 17:29:27.511626 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:27.511596 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsdff" Nov 02 17:29:27.512058 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:27.511627 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:27.512058 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:27.511627 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:27.512058 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:27.511731 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsdff" podUID="d09569ff-f9b3-42d4-b163-fae46612bffc" Nov 02 17:29:27.512058 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:27.511793 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2q87" podUID="83c827e3-5338-49b5-97a1-56bdb5904500" Nov 02 17:29:27.512058 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:27.511851 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdtgx" podUID="5cb013c0-4887-488b-867f-841fb6b01ca5" Nov 02 17:29:29.511719 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:29.511681 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:29.512215 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:29.511687 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:29.512215 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:29.511788 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2q87" podUID="83c827e3-5338-49b5-97a1-56bdb5904500" Nov 02 17:29:29.512215 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:29.511687 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsdff" Nov 02 17:29:29.512215 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:29.511870 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdtgx" podUID="5cb013c0-4887-488b-867f-841fb6b01ca5" Nov 02 17:29:29.512215 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:29.511962 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsdff" podUID="d09569ff-f9b3-42d4-b163-fae46612bffc" Nov 02 17:29:31.182114 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.182084 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-26.ec2.internal" event="NodeReady" Nov 02 17:29:31.182561 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.182208 2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 17:29:31.281825 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.279504 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bkdn9"] Nov 02 17:29:31.282925 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.282899 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bkdn9" Nov 02 17:29:31.287474 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.287449 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cqljg"] Nov 02 17:29:31.290213 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.290192 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cqljg" Nov 02 17:29:31.301412 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.301394 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 17:29:31.301530 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.301432 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 17:29:31.301530 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.301447 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5nstj\"" Nov 02 17:29:31.313393 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.313370 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bkdn9"] Nov 02 17:29:31.335747 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.335724 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cqljg"] Nov 02 17:29:31.403456 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.403420 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 17:29:31.403594 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.403498 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rj9zp\"" Nov 02 17:29:31.404396 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.404377 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 17:29:31.404396 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.404386 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 17:29:31.443605 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.443584 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmgzk\" (UniqueName: \"kubernetes.io/projected/d911b628-4ba2-4427-b229-7698d25c2e2c-kube-api-access-dmgzk\") pod \"ingress-canary-cqljg\" (UID: \"d911b628-4ba2-4427-b229-7698d25c2e2c\") " pod="openshift-ingress-canary/ingress-canary-cqljg" Nov 02 17:29:31.443679 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.443614 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d601642-0fa6-4b68-b9cf-324c59e41637-tmp-dir\") pod \"dns-default-bkdn9\" (UID: \"4d601642-0fa6-4b68-b9cf-324c59e41637\") " pod="openshift-dns/dns-default-bkdn9" Nov 02 17:29:31.443679 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.443634 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d911b628-4ba2-4427-b229-7698d25c2e2c-cert\") pod \"ingress-canary-cqljg\" (UID: \"d911b628-4ba2-4427-b229-7698d25c2e2c\") " pod="openshift-ingress-canary/ingress-canary-cqljg" Nov 02 17:29:31.443679 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.443669 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4d601642-0fa6-4b68-b9cf-324c59e41637-config-volume\") pod \"dns-default-bkdn9\" (UID: \"4d601642-0fa6-4b68-b9cf-324c59e41637\") " pod="openshift-dns/dns-default-bkdn9" Nov 02 17:29:31.443796 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.443746 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvnsx\" (UniqueName: \"kubernetes.io/projected/4d601642-0fa6-4b68-b9cf-324c59e41637-kube-api-access-wvnsx\") pod \"dns-default-bkdn9\" (UID: \"4d601642-0fa6-4b68-b9cf-324c59e41637\") " pod="openshift-dns/dns-default-bkdn9" Nov 02 17:29:31.443851 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.443831 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d601642-0fa6-4b68-b9cf-324c59e41637-metrics-tls\") pod \"dns-default-bkdn9\" (UID: \"4d601642-0fa6-4b68-b9cf-324c59e41637\") " pod="openshift-dns/dns-default-bkdn9" Nov 02 17:29:31.511909 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.511880 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:31.511909 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.511903 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsdff" Nov 02 17:29:31.512058 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.512048 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:31.525695 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.525678 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 17:29:31.525807 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.525792 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 17:29:31.525896 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.525832 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 17:29:31.525896 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.525846 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nn2p2\"" Nov 02 17:29:31.526058 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.526040 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zt2z5\"" Nov 02 17:29:31.526142 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.526069 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 17:29:31.526142 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.526084 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ppwdb\"" Nov 02 17:29:31.544834 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.544782 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4d601642-0fa6-4b68-b9cf-324c59e41637-config-volume\") pod \"dns-default-bkdn9\" (UID: \"4d601642-0fa6-4b68-b9cf-324c59e41637\") " pod="openshift-dns/dns-default-bkdn9" Nov 02 17:29:31.544834 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.544823 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvnsx\" (UniqueName: \"kubernetes.io/projected/4d601642-0fa6-4b68-b9cf-324c59e41637-kube-api-access-wvnsx\") pod \"dns-default-bkdn9\" (UID: \"4d601642-0fa6-4b68-b9cf-324c59e41637\") " pod="openshift-dns/dns-default-bkdn9" Nov 02 17:29:31.544996 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.544902 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d601642-0fa6-4b68-b9cf-324c59e41637-metrics-tls\") pod \"dns-default-bkdn9\" (UID: \"4d601642-0fa6-4b68-b9cf-324c59e41637\") " pod="openshift-dns/dns-default-bkdn9" Nov 02 17:29:31.544996 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.544929 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmgzk\" (UniqueName: \"kubernetes.io/projected/d911b628-4ba2-4427-b229-7698d25c2e2c-kube-api-access-dmgzk\") pod \"ingress-canary-cqljg\" (UID: \"d911b628-4ba2-4427-b229-7698d25c2e2c\") " pod="openshift-ingress-canary/ingress-canary-cqljg" Nov 02 17:29:31.544996 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.544966 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d601642-0fa6-4b68-b9cf-324c59e41637-tmp-dir\") pod \"dns-default-bkdn9\" (UID: \"4d601642-0fa6-4b68-b9cf-324c59e41637\") " pod="openshift-dns/dns-default-bkdn9" Nov 02 17:29:31.544996 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.544993 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d911b628-4ba2-4427-b229-7698d25c2e2c-cert\") pod \"ingress-canary-cqljg\" (UID: \"d911b628-4ba2-4427-b229-7698d25c2e2c\") " pod="openshift-ingress-canary/ingress-canary-cqljg" Nov 02 17:29:31.545154 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:31.545137 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 17:29:31.545219 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:31.545208 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d601642-0fa6-4b68-b9cf-324c59e41637-metrics-tls podName:4d601642-0fa6-4b68-b9cf-324c59e41637 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:32.0451827 +0000 UTC m=+30.112728492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4d601642-0fa6-4b68-b9cf-324c59e41637-metrics-tls") pod "dns-default-bkdn9" (UID: "4d601642-0fa6-4b68-b9cf-324c59e41637") : secret "dns-default-metrics-tls" not found Nov 02 17:29:31.545596 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.545578 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d601642-0fa6-4b68-b9cf-324c59e41637-tmp-dir\") pod \"dns-default-bkdn9\" (UID: \"4d601642-0fa6-4b68-b9cf-324c59e41637\") " pod="openshift-dns/dns-default-bkdn9" Nov 02 17:29:31.545695 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.545676 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4d601642-0fa6-4b68-b9cf-324c59e41637-config-volume\") pod \"dns-default-bkdn9\" (UID: \"4d601642-0fa6-4b68-b9cf-324c59e41637\") " pod="openshift-dns/dns-default-bkdn9" Nov 02 17:29:31.549008 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.548892 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d911b628-4ba2-4427-b229-7698d25c2e2c-cert\") pod \"ingress-canary-cqljg\" (UID: \"d911b628-4ba2-4427-b229-7698d25c2e2c\") " pod="openshift-ingress-canary/ingress-canary-cqljg" Nov 02 17:29:31.569223 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.569201 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvnsx\" (UniqueName: \"kubernetes.io/projected/4d601642-0fa6-4b68-b9cf-324c59e41637-kube-api-access-wvnsx\") pod \"dns-default-bkdn9\" (UID: \"4d601642-0fa6-4b68-b9cf-324c59e41637\") " pod="openshift-dns/dns-default-bkdn9" Nov 02 17:29:31.569322 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.569291 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmgzk\" (UniqueName: \"kubernetes.io/projected/d911b628-4ba2-4427-b229-7698d25c2e2c-kube-api-access-dmgzk\") pod \"ingress-canary-cqljg\" (UID: \"d911b628-4ba2-4427-b229-7698d25c2e2c\") " pod="openshift-ingress-canary/ingress-canary-cqljg" Nov 02 17:29:31.601174 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.601149 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cqljg" Nov 02 17:29:31.910457 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:31.910386 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cqljg"] Nov 02 17:29:31.914728 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:31.914701 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd911b628_4ba2_4427_b229_7698d25c2e2c.slice/crio-233c9f1f93bfdf7c1b0ed3b4e671557d2b6d78d24d586f3c0cd7ea694c5ab72d WatchSource:0}: Error finding container 233c9f1f93bfdf7c1b0ed3b4e671557d2b6d78d24d586f3c0cd7ea694c5ab72d: Status 404 returned error can't find the container with id 233c9f1f93bfdf7c1b0ed3b4e671557d2b6d78d24d586f3c0cd7ea694c5ab72d Nov 02 17:29:32.048128 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:32.048095 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d601642-0fa6-4b68-b9cf-324c59e41637-metrics-tls\") pod \"dns-default-bkdn9\" (UID: \"4d601642-0fa6-4b68-b9cf-324c59e41637\") " pod="openshift-dns/dns-default-bkdn9" Nov 02 17:29:32.048330 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:32.048210 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 17:29:32.048330 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:32.048270 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d601642-0fa6-4b68-b9cf-324c59e41637-metrics-tls podName:4d601642-0fa6-4b68-b9cf-324c59e41637 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:33.04825765 +0000 UTC m=+31.115803407 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4d601642-0fa6-4b68-b9cf-324c59e41637-metrics-tls") pod "dns-default-bkdn9" (UID: "4d601642-0fa6-4b68-b9cf-324c59e41637") : secret "dns-default-metrics-tls" not found Nov 02 17:29:32.771544 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:32.771499 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cqljg" event={"ID":"d911b628-4ba2-4427-b229-7698d25c2e2c","Type":"ContainerStarted","Data":"233c9f1f93bfdf7c1b0ed3b4e671557d2b6d78d24d586f3c0cd7ea694c5ab72d"} Nov 02 17:29:33.057190 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:33.057105 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d601642-0fa6-4b68-b9cf-324c59e41637-metrics-tls\") pod \"dns-default-bkdn9\" (UID: \"4d601642-0fa6-4b68-b9cf-324c59e41637\") " pod="openshift-dns/dns-default-bkdn9" Nov 02 17:29:33.057366 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:33.057265 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 17:29:33.057366 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:33.057345 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d601642-0fa6-4b68-b9cf-324c59e41637-metrics-tls podName:4d601642-0fa6-4b68-b9cf-324c59e41637 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:35.057327004 +0000 UTC m=+33.124872781 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4d601642-0fa6-4b68-b9cf-324c59e41637-metrics-tls") pod "dns-default-bkdn9" (UID: "4d601642-0fa6-4b68-b9cf-324c59e41637") : secret "dns-default-metrics-tls" not found Nov 02 17:29:35.073135 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:35.073087 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d601642-0fa6-4b68-b9cf-324c59e41637-metrics-tls\") pod \"dns-default-bkdn9\" (UID: \"4d601642-0fa6-4b68-b9cf-324c59e41637\") " pod="openshift-dns/dns-default-bkdn9" Nov 02 17:29:35.073649 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:35.073244 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 17:29:35.073649 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:35.073332 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d601642-0fa6-4b68-b9cf-324c59e41637-metrics-tls podName:4d601642-0fa6-4b68-b9cf-324c59e41637 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:39.073298256 +0000 UTC m=+37.140844015 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4d601642-0fa6-4b68-b9cf-324c59e41637-metrics-tls") pod "dns-default-bkdn9" (UID: "4d601642-0fa6-4b68-b9cf-324c59e41637") : secret "dns-default-metrics-tls" not found Nov 02 17:29:35.780708 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:35.780638 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cqljg" event={"ID":"d911b628-4ba2-4427-b229-7698d25c2e2c","Type":"ContainerStarted","Data":"b29c7ee3fc03c22d04fd75f5c39a7e088ca4aecef28b7fa787b49406e685ca41"} Nov 02 17:29:35.832206 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:35.832160 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cqljg" podStartSLOduration=1.904690602 podStartE2EDuration="4.83214346s" totalImagesPullingTime="2.927452858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:29:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:35.831676453 +0000 UTC m=+33.899222232" watchObservedRunningTime="2026-11-02 17:29:35.83214346 +0000 UTC m=+33.899689241" Nov 02 17:29:36.179834 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:36.179745 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cb013c0-4887-488b-867f-841fb6b01ca5-metrics-certs\") pod \"network-metrics-daemon-fdtgx\" (UID: \"5cb013c0-4887-488b-867f-841fb6b01ca5\") " pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:36.179834 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:36.179816 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83c827e3-5338-49b5-97a1-56bdb5904500-original-pull-secret\") pod \"global-pull-secret-syncer-l2q87\" (UID: \"83c827e3-5338-49b5-97a1-56bdb5904500\") " pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:36.180372 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:36.179919 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 17:29:36.180372 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:36.180003 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cb013c0-4887-488b-867f-841fb6b01ca5-metrics-certs podName:5cb013c0-4887-488b-867f-841fb6b01ca5 nodeName:}" failed. No retries permitted until 2026-11-02 17:30:08.179981643 +0000 UTC m=+66.247527416 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5cb013c0-4887-488b-867f-841fb6b01ca5-metrics-certs") pod "network-metrics-daemon-fdtgx" (UID: "5cb013c0-4887-488b-867f-841fb6b01ca5") : secret "metrics-daemon-secret" not found Nov 02 17:29:36.182396 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:36.182372 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83c827e3-5338-49b5-97a1-56bdb5904500-original-pull-secret\") pod \"global-pull-secret-syncer-l2q87\" (UID: \"83c827e3-5338-49b5-97a1-56bdb5904500\") " pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:36.280707 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:36.280667 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv9jt\" (UniqueName: \"kubernetes.io/projected/d09569ff-f9b3-42d4-b163-fae46612bffc-kube-api-access-cv9jt\") pod \"network-check-target-hsdff\" (UID: \"d09569ff-f9b3-42d4-b163-fae46612bffc\") " pod="openshift-network-diagnostics/network-check-target-hsdff" Nov 02 17:29:36.283205 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:36.283182 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv9jt\" (UniqueName: \"kubernetes.io/projected/d09569ff-f9b3-42d4-b163-fae46612bffc-kube-api-access-cv9jt\") pod \"network-check-target-hsdff\" (UID: \"d09569ff-f9b3-42d4-b163-fae46612bffc\") " pod="openshift-network-diagnostics/network-check-target-hsdff" Nov 02 17:29:36.328978 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:36.328951 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:36.334103 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:36.334083 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsdff" Nov 02 17:29:36.693408 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:36.693376 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l2q87"] Nov 02 17:29:36.696650 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:36.696620 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c827e3_5338_49b5_97a1_56bdb5904500.slice/crio-42ca45cc6abd1e597a513f448d8d64c0af835aaaa024e1559332ae203fb2799a WatchSource:0}: Error finding container 42ca45cc6abd1e597a513f448d8d64c0af835aaaa024e1559332ae203fb2799a: Status 404 returned error can't find the container with id 42ca45cc6abd1e597a513f448d8d64c0af835aaaa024e1559332ae203fb2799a Nov 02 17:29:36.726519 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:36.726452 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hsdff"] Nov 02 17:29:36.729521 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:36.729497 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd09569ff_f9b3_42d4_b163_fae46612bffc.slice/crio-8a67aea3907d74aacfa4e7d623918e1e259725faab454313457660e49c28e754 WatchSource:0}: Error finding container 8a67aea3907d74aacfa4e7d623918e1e259725faab454313457660e49c28e754: Status 404 returned error can't find the container with id 8a67aea3907d74aacfa4e7d623918e1e259725faab454313457660e49c28e754 Nov 02 17:29:36.783532 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:36.783487 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hsdff" event={"ID":"d09569ff-f9b3-42d4-b163-fae46612bffc","Type":"ContainerStarted","Data":"8a67aea3907d74aacfa4e7d623918e1e259725faab454313457660e49c28e754"} Nov 02 17:29:36.784645 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:36.784620 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l2q87" event={"ID":"83c827e3-5338-49b5-97a1-56bdb5904500","Type":"ContainerStarted","Data":"42ca45cc6abd1e597a513f448d8d64c0af835aaaa024e1559332ae203fb2799a"} Nov 02 17:29:37.564032 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:37.563995 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ccwcv"] Nov 02 17:29:37.584745 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:37.584717 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:37.599289 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:37.599263 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ccwcv"] Nov 02 17:29:37.612481 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:37.612453 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fw7vt\"" Nov 02 17:29:37.623254 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:37.623224 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 17:29:37.623643 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:37.623364 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 17:29:37.623643 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:37.623553 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 17:29:37.623643 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:37.623615 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 17:29:37.690319 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:37.690269 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/007d7841-411c-4b0f-b8aa-d36097b6bf31-crio-socket\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:37.690489 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:37.690336 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/007d7841-411c-4b0f-b8aa-d36097b6bf31-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:37.690489 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:37.690368 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmjd4\" (UniqueName: \"kubernetes.io/projected/007d7841-411c-4b0f-b8aa-d36097b6bf31-kube-api-access-pmjd4\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:37.690489 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:37.690410 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/007d7841-411c-4b0f-b8aa-d36097b6bf31-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:37.690489 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:37.690454 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/007d7841-411c-4b0f-b8aa-d36097b6bf31-data-volume\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:37.791336 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:37.790975 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/007d7841-411c-4b0f-b8aa-d36097b6bf31-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:37.791336 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:37.791076 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/007d7841-411c-4b0f-b8aa-d36097b6bf31-data-volume\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:37.791336 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:37.791141 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 17:29:37.791336 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:37.791155 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/007d7841-411c-4b0f-b8aa-d36097b6bf31-crio-socket\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:37.791336 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:37.791190 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/007d7841-411c-4b0f-b8aa-d36097b6bf31-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:37.791336 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:37.791218 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/007d7841-411c-4b0f-b8aa-d36097b6bf31-insights-runtime-extractor-tls podName:007d7841-411c-4b0f-b8aa-d36097b6bf31 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:38.291196012 +0000 UTC m=+36.358741772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/007d7841-411c-4b0f-b8aa-d36097b6bf31-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ccwcv" (UID: "007d7841-411c-4b0f-b8aa-d36097b6bf31") : secret "insights-runtime-extractor-tls" not found Nov 02 17:29:37.791336 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:37.791249 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 17:29:37.791336 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:37.791256 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmjd4\" (UniqueName: \"kubernetes.io/projected/007d7841-411c-4b0f-b8aa-d36097b6bf31-kube-api-access-pmjd4\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:37.791737 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:37.791462 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/007d7841-411c-4b0f-b8aa-d36097b6bf31-data-volume\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:37.791737 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:37.791652 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/007d7841-411c-4b0f-b8aa-d36097b6bf31-crio-socket\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:37.791867 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:37.791843 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/007d7841-411c-4b0f-b8aa-d36097b6bf31-kube-rbac-proxy-cm podName:007d7841-411c-4b0f-b8aa-d36097b6bf31 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:38.291291831 +0000 UTC m=+36.358837607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/007d7841-411c-4b0f-b8aa-d36097b6bf31-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ccwcv" (UID: "007d7841-411c-4b0f-b8aa-d36097b6bf31") : configmap "kube-rbac-proxy" not found Nov 02 17:29:37.816196 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:37.816136 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmjd4\" (UniqueName: \"kubernetes.io/projected/007d7841-411c-4b0f-b8aa-d36097b6bf31-kube-api-access-pmjd4\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:38.295845 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:38.295801 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/007d7841-411c-4b0f-b8aa-d36097b6bf31-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:38.296027 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:38.295854 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/007d7841-411c-4b0f-b8aa-d36097b6bf31-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:38.296027 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:38.295970 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 17:29:38.296145 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:38.296035 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/007d7841-411c-4b0f-b8aa-d36097b6bf31-insights-runtime-extractor-tls podName:007d7841-411c-4b0f-b8aa-d36097b6bf31 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:39.296016273 +0000 UTC m=+37.363562036 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/007d7841-411c-4b0f-b8aa-d36097b6bf31-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ccwcv" (UID: "007d7841-411c-4b0f-b8aa-d36097b6bf31") : secret "insights-runtime-extractor-tls" not found Nov 02 17:29:38.296558 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:38.296466 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 17:29:38.296558 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:38.296524 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/007d7841-411c-4b0f-b8aa-d36097b6bf31-kube-rbac-proxy-cm podName:007d7841-411c-4b0f-b8aa-d36097b6bf31 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:39.296509416 +0000 UTC m=+37.364055190 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/007d7841-411c-4b0f-b8aa-d36097b6bf31-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ccwcv" (UID: "007d7841-411c-4b0f-b8aa-d36097b6bf31") : configmap "kube-rbac-proxy" not found Nov 02 17:29:38.320613 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:38.320580 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-bffdk"] Nov 02 17:29:38.335875 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:38.335846 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bffdk" Nov 02 17:29:38.347785 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:38.347757 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-bffdk"] Nov 02 17:29:38.359273 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:38.359251 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 17:29:38.359471 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:38.359453 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-vqf9q\"" Nov 02 17:29:38.359556 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:38.359543 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 17:29:38.498167 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:38.498129 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qcnl\" (UniqueName: \"kubernetes.io/projected/3a4bf99d-1b04-48a4-bca9-dd3f484ad0a8-kube-api-access-4qcnl\") pod \"migrator-66df5d496b-bffdk\" (UID: \"3a4bf99d-1b04-48a4-bca9-dd3f484ad0a8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bffdk" Nov 02 17:29:38.598754 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:38.598672 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qcnl\" (UniqueName: \"kubernetes.io/projected/3a4bf99d-1b04-48a4-bca9-dd3f484ad0a8-kube-api-access-4qcnl\") pod \"migrator-66df5d496b-bffdk\" (UID: \"3a4bf99d-1b04-48a4-bca9-dd3f484ad0a8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bffdk" Nov 02 17:29:38.622888 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:38.622822 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qcnl\" (UniqueName: \"kubernetes.io/projected/3a4bf99d-1b04-48a4-bca9-dd3f484ad0a8-kube-api-access-4qcnl\") pod \"migrator-66df5d496b-bffdk\" (UID: \"3a4bf99d-1b04-48a4-bca9-dd3f484ad0a8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bffdk" Nov 02 17:29:38.662814 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:38.662600 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bffdk" Nov 02 17:29:38.911289 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:38.911208 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-mpdt5"] Nov 02 17:29:38.936777 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:38.936746 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-mpdt5" Nov 02 17:29:38.937006 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:38.936795 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-mpdt5"] Nov 02 17:29:38.951444 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:38.951367 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 02 17:29:38.961446 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:38.961422 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-w9x92\"" Nov 02 17:29:38.961579 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:38.961451 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 02 17:29:38.961649 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:38.961583 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 02 17:29:38.961701 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:38.961665 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 02 17:29:38.991673 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:38.991645 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-bffdk"] Nov 02 17:29:38.995296 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:38.995256 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a4bf99d_1b04_48a4_bca9_dd3f484ad0a8.slice/crio-7fa25d80d54565f9834c98d65add99ab446169d815d3234a2f109707ff8b1893 WatchSource:0}: Error finding container 7fa25d80d54565f9834c98d65add99ab446169d815d3234a2f109707ff8b1893: Status 404 returned error can't find the container with id 7fa25d80d54565f9834c98d65add99ab446169d815d3234a2f109707ff8b1893 Nov 02 17:29:39.104398 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:39.103671 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/81019158-9e01-4dd1-a405-d1cf75d6dc78-signing-cabundle\") pod \"service-ca-5fffc97bdf-mpdt5\" (UID: \"81019158-9e01-4dd1-a405-d1cf75d6dc78\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mpdt5" Nov 02 17:29:39.104398 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:39.103732 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/81019158-9e01-4dd1-a405-d1cf75d6dc78-signing-key\") pod \"service-ca-5fffc97bdf-mpdt5\" (UID: \"81019158-9e01-4dd1-a405-d1cf75d6dc78\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mpdt5" Nov 02 17:29:39.104398 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:39.103765 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk945\" (UniqueName: \"kubernetes.io/projected/81019158-9e01-4dd1-a405-d1cf75d6dc78-kube-api-access-pk945\") pod \"service-ca-5fffc97bdf-mpdt5\" (UID: \"81019158-9e01-4dd1-a405-d1cf75d6dc78\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mpdt5" Nov 02 17:29:39.104398 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:39.103841 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d601642-0fa6-4b68-b9cf-324c59e41637-metrics-tls\") pod \"dns-default-bkdn9\" (UID: \"4d601642-0fa6-4b68-b9cf-324c59e41637\") " pod="openshift-dns/dns-default-bkdn9" Nov 02 17:29:39.104398 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:39.103949 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 17:29:39.104398 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:39.103995 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d601642-0fa6-4b68-b9cf-324c59e41637-metrics-tls podName:4d601642-0fa6-4b68-b9cf-324c59e41637 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:47.103982866 +0000 UTC m=+45.171528638 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4d601642-0fa6-4b68-b9cf-324c59e41637-metrics-tls") pod "dns-default-bkdn9" (UID: "4d601642-0fa6-4b68-b9cf-324c59e41637") : secret "dns-default-metrics-tls" not found Nov 02 17:29:39.204997 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:39.204919 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/81019158-9e01-4dd1-a405-d1cf75d6dc78-signing-cabundle\") pod \"service-ca-5fffc97bdf-mpdt5\" (UID: \"81019158-9e01-4dd1-a405-d1cf75d6dc78\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mpdt5" Nov 02 17:29:39.204997 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:39.204992 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/81019158-9e01-4dd1-a405-d1cf75d6dc78-signing-key\") pod \"service-ca-5fffc97bdf-mpdt5\" (UID: \"81019158-9e01-4dd1-a405-d1cf75d6dc78\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mpdt5" Nov 02 17:29:39.205237 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:39.205022 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk945\" (UniqueName: \"kubernetes.io/projected/81019158-9e01-4dd1-a405-d1cf75d6dc78-kube-api-access-pk945\") pod \"service-ca-5fffc97bdf-mpdt5\" (UID: \"81019158-9e01-4dd1-a405-d1cf75d6dc78\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mpdt5" Nov 02 17:29:39.205805 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:39.205741 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/81019158-9e01-4dd1-a405-d1cf75d6dc78-signing-cabundle\") pod \"service-ca-5fffc97bdf-mpdt5\" (UID: \"81019158-9e01-4dd1-a405-d1cf75d6dc78\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mpdt5" Nov 02 17:29:39.208001 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:39.207973 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/81019158-9e01-4dd1-a405-d1cf75d6dc78-signing-key\") pod \"service-ca-5fffc97bdf-mpdt5\" (UID: \"81019158-9e01-4dd1-a405-d1cf75d6dc78\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mpdt5" Nov 02 17:29:39.242937 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:39.242905 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk945\" (UniqueName: \"kubernetes.io/projected/81019158-9e01-4dd1-a405-d1cf75d6dc78-kube-api-access-pk945\") pod \"service-ca-5fffc97bdf-mpdt5\" (UID: \"81019158-9e01-4dd1-a405-d1cf75d6dc78\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mpdt5" Nov 02 17:29:39.271708 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:39.271677 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-mpdt5" Nov 02 17:29:39.306402 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:39.305751 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/007d7841-411c-4b0f-b8aa-d36097b6bf31-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:39.306402 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:39.305836 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/007d7841-411c-4b0f-b8aa-d36097b6bf31-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:39.306402 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:39.305910 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 17:29:39.306402 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:39.305967 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/007d7841-411c-4b0f-b8aa-d36097b6bf31-kube-rbac-proxy-cm podName:007d7841-411c-4b0f-b8aa-d36097b6bf31 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:41.305951297 +0000 UTC m=+39.373497069 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/007d7841-411c-4b0f-b8aa-d36097b6bf31-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ccwcv" (UID: "007d7841-411c-4b0f-b8aa-d36097b6bf31") : configmap "kube-rbac-proxy" not found Nov 02 17:29:39.306402 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:39.306332 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 17:29:39.306402 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:39.306369 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/007d7841-411c-4b0f-b8aa-d36097b6bf31-insights-runtime-extractor-tls podName:007d7841-411c-4b0f-b8aa-d36097b6bf31 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:41.306359545 +0000 UTC m=+39.373905304 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/007d7841-411c-4b0f-b8aa-d36097b6bf31-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ccwcv" (UID: "007d7841-411c-4b0f-b8aa-d36097b6bf31") : secret "insights-runtime-extractor-tls" not found Nov 02 17:29:39.592101 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:39.592063 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-mpdt5"] Nov 02 17:29:39.596875 ip-10-0-142-26 kubenswrapper[2580]: W1102 17:29:39.596841 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81019158_9e01_4dd1_a405_d1cf75d6dc78.slice/crio-faa537da1489a54bb0d2bfde98e63d93a68abf0c1eb1b471055ada636900af82 WatchSource:0}: Error finding container faa537da1489a54bb0d2bfde98e63d93a68abf0c1eb1b471055ada636900af82: Status 404 returned error can't find the container with id faa537da1489a54bb0d2bfde98e63d93a68abf0c1eb1b471055ada636900af82 Nov 02 17:29:39.794830 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:39.794791 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-mpdt5" event={"ID":"81019158-9e01-4dd1-a405-d1cf75d6dc78","Type":"ContainerStarted","Data":"faa537da1489a54bb0d2bfde98e63d93a68abf0c1eb1b471055ada636900af82"} Nov 02 17:29:39.796566 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:39.796537 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bffdk" event={"ID":"3a4bf99d-1b04-48a4-bca9-dd3f484ad0a8","Type":"ContainerStarted","Data":"7fa25d80d54565f9834c98d65add99ab446169d815d3234a2f109707ff8b1893"} Nov 02 17:29:40.802224 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:40.801781 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hsdff" event={"ID":"d09569ff-f9b3-42d4-b163-fae46612bffc","Type":"ContainerStarted","Data":"610ea65ab50f4a94575df495dc8624297ed43f89e755cde06df8a9ea7b31377f"} Nov 02 17:29:40.802224 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:40.802057 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hsdff" Nov 02 17:29:40.804539 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:40.804503 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bffdk" event={"ID":"3a4bf99d-1b04-48a4-bca9-dd3f484ad0a8","Type":"ContainerStarted","Data":"93982355ad479f9f88e7ee70f7abb4b5a70e1b27cc823fdb444d7ea17933c1cb"} Nov 02 17:29:40.839890 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:40.839834 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hsdff" podStartSLOduration=35.372621625 podStartE2EDuration="38.839817434s" totalImagesPullingTime="3.467195809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:40.839204714 +0000 UTC m=+38.906750495" watchObservedRunningTime="2026-11-02 17:29:40.839817434 +0000 UTC m=+38.907363217" Nov 02 17:29:40.877601 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:40.877545 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bffdk" podStartSLOduration=1.451391832 podStartE2EDuration="2.877524549s" totalImagesPullingTime="1.426132717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:40.87627385 +0000 UTC m=+38.943819632" watchObservedRunningTime="2026-11-02 17:29:40.877524549 +0000 UTC m=+38.945070330" Nov 02 17:29:41.319109 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:41.319066 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/007d7841-411c-4b0f-b8aa-d36097b6bf31-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:41.319330 ip-10-0-142-26 kubenswrapper[2580]: I1102 17:29:41.319126 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/007d7841-411c-4b0f-b8aa-d36097b6bf31-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:41.319330 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:41.319213 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 17:29:41.319330 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:41.319277 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 17:29:41.319330 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:41.319299 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/007d7841-411c-4b0f-b8aa-d36097b6bf31-kube-rbac-proxy-cm podName:007d7841-411c-4b0f-b8aa-d36097b6bf31 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:45.319276493 +0000 UTC m=+43.386822267 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/007d7841-411c-4b0f-b8aa-d36097b6bf31-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ccwcv" (UID: "007d7841-411c-4b0f-b8aa-d36097b6bf31") : configmap "kube-rbac-proxy" not found Nov 02 17:29:41.319522 ip-10-0-142-26 kubenswrapper[2580]: E1102 17:29:41.319346 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/007d7841-411c-4b0f-b8aa-d36097b6bf31-insights-runtime-extractor-tls podName:007d7841-411c-4b0f-b8aa-d36097b6bf31 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:45.319327494 +0000 UTC m=+43.386873272 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/007d7841-411c-4b0f-b8aa-d36097b6bf31-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ccwcv" (UID: "007d7841-411c-4b0f-b8aa-d36097b6bf31") : secret "insights-runtime-extractor-tls" not found Nov 02 17:29:42.229480 ip-10-0-142-26 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 17:29:42.241673 ip-10-0-142-26 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 17:29:42.241842 ip-10-0-142-26 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 17:29:42.241913 ip-10-0-142-26 systemd[1]: kubelet.service: Consumed 2.273s CPU time. Nov 02 17:29:42.258782 ip-10-0-142-26 systemd[1]: Starting Kubernetes Kubelet... Nov 02 17:29:42.352704 ip-10-0-142-26 kubenswrapper[4471]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:29:42.352704 ip-10-0-142-26 kubenswrapper[4471]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 17:29:42.352704 ip-10-0-142-26 kubenswrapper[4471]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:29:42.352704 ip-10-0-142-26 kubenswrapper[4471]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:29:42.354757 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354678 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:29:42.354757 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354756 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:29:42.354757 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354761 4471 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:29:42.354860 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354765 4471 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:29:42.354860 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354768 4471 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:29:42.354860 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354771 4471 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:29:42.354860 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354774 4471 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:29:42.354860 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354777 4471 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:29:42.354860 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354780 4471 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:29:42.354860 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354783 4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:29:42.354860 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354785 4471 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:29:42.354860 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354788 4471 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:29:42.354860 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354791 4471 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:29:42.354860 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354793 4471 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:29:42.354860 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354796 4471 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:29:42.354860 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354799 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:29:42.354860 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354802 4471 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:29:42.354860 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354805 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:29:42.354860 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354807 4471 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:29:42.354860 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354813 4471 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:29:42.354860 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354816 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:29:42.354860 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354819 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:29:42.354860 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354821 4471 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:29:42.355375 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354824 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:29:42.355375 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354827 4471 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:29:42.355375 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354832 4471 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:29:42.355375 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354836 4471 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:29:42.355375 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354839 4471 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:29:42.355375 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354842 4471 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:29:42.355375 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354845 4471 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:29:42.355375 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354847 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:29:42.355375 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354850 4471 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:29:42.355375 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354854 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:29:42.355375 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354857 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:29:42.355375 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354860 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:29:42.355375 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354863 4471 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:29:42.355375 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354865 4471 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:29:42.355375 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354868 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:29:42.355375 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354871 4471 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:29:42.355375 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354874 4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:29:42.355375 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354876 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:29:42.355375 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354879 4471 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:29:42.355852 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354882 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:29:42.355852 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354885 4471 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:29:42.355852 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354887 4471 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:29:42.355852 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354890 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:29:42.355852 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354893 4471 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:29:42.355852 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354895 4471 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:29:42.355852 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354898 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:29:42.355852 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354901 4471 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:29:42.355852 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354903 4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:29:42.355852 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354906 4471 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:29:42.355852 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354909 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:29:42.355852 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354912 4471 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:29:42.355852 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354914 4471 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:29:42.355852 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354917 4471 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:29:42.355852 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354919 4471 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:29:42.355852 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354922 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:29:42.355852 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354925 4471 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:29:42.355852 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354930 4471 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:29:42.355852 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354933 4471 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:29:42.355852 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354937 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:29:42.356396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354940 4471 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:29:42.356396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354942 4471 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:29:42.356396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354945 4471 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:29:42.356396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354948 4471 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:29:42.356396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354950 4471 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:29:42.356396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354953 4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:29:42.356396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354955 4471 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:29:42.356396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354958 4471 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:29:42.356396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354961 4471 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:29:42.356396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354963 4471 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:29:42.356396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354967 4471 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:29:42.356396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354970 4471 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:29:42.356396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354973 4471 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:29:42.356396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354976 4471 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:29:42.356396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354979 4471 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:29:42.356396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354981 4471 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:29:42.356396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354984 4471 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:29:42.356396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354986 4471 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:29:42.356396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354989 4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:29:42.356396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354991 4471 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:29:42.356890 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354994 4471 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:29:42.356890 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354997 4471 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:29:42.356890 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.354999 4471 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:29:42.356890 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355003 4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:29:42.356890 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355006 4471 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:29:42.356890 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355008 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:29:42.356890 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355011 4471 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:29:42.356890 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355014 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:29:42.356890 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355016 4471 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:29:42.356890 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355019 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:29:42.356890 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355021 4471 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:29:42.356890 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355024 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:29:42.356890 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355026 4471 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:29:42.356890 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355030 4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:29:42.356890 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355033 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:29:42.356890 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355035 4471 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:29:42.356890 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355039 4471 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:29:42.356890 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355041 4471 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:29:42.356890 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355044 4471 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:29:42.357377 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355047 4471 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:29:42.357377 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355049 4471 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:29:42.357377 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355052 4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:29:42.357377 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355054 4471 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:29:42.357377 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355057 4471 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:29:42.357377 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355833 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:29:42.357377 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355841 4471 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:29:42.357377 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355845 4471 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:29:42.357377 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355848 4471 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:29:42.357377 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355851 4471 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:29:42.357377 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355854 4471 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:29:42.357377 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355857 4471 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:29:42.357377 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355860 4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:29:42.357377 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355863 4471 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:29:42.357377 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355866 4471 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:29:42.357377 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355868 4471 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:29:42.357377 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355871 4471 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:29:42.357377 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355874 4471 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:29:42.357377 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355876 4471 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:29:42.357377 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355879 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:29:42.357916 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355882 4471 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:29:42.357916 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355885 4471 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:29:42.357916 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355887 4471 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:29:42.357916 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355890 4471 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:29:42.357916 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355892 4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:29:42.357916 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355895 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:29:42.357916 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355897 4471 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:29:42.357916 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355900 4471 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:29:42.357916 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355903 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:29:42.357916 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355905 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:29:42.357916 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355909 4471 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:29:42.357916 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355912 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:29:42.357916 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355914 4471 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:29:42.357916 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355917 4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:29:42.357916 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355919 4471 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:29:42.357916 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355922 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:29:42.357916 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355924 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:29:42.357916 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355928 4471 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:29:42.357916 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355931 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:29:42.358405 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355933 4471 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:29:42.358405 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355936 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:29:42.358405 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355938 4471 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:29:42.358405 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355941 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:29:42.358405 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355943 4471 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:29:42.358405 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355946 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:29:42.358405 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355948 4471 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:29:42.358405 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355951 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:29:42.358405 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355954 4471 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:29:42.358405 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355956 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:29:42.358405 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355959 4471 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:29:42.358405 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355961 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:29:42.358405 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355964 4471 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:29:42.358405 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355966 4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:29:42.358405 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355970 4471 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:29:42.358405 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355973 4471 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:29:42.358405 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355976 4471 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:29:42.358405 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355979 4471 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:29:42.358405 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355982 4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:29:42.358887 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355985 4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:29:42.358887 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355987 4471 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:29:42.358887 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.355990 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:29:42.358887 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356009 4471 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:29:42.358887 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356013 4471 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:29:42.358887 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356016 4471 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:29:42.358887 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356019 4471 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:29:42.358887 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356022 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:29:42.358887 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356024 4471 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:29:42.358887 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356027 4471 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:29:42.358887 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356030 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:29:42.358887 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356032 4471 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:29:42.358887 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356035 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:29:42.358887 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356037 4471 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:29:42.358887 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356040 4471 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:29:42.358887 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356043 4471 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:29:42.358887 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356045 4471 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:29:42.358887 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356049 4471 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:29:42.358887 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356054 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:29:42.359373 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356057 4471 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:29:42.359373 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356059 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:29:42.359373 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356062 4471 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:29:42.359373 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356065 4471 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:29:42.359373 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356068 4471 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:29:42.359373 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356071 4471 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:29:42.359373 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356073 4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:29:42.359373 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356076 4471 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:29:42.359373 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356079 4471 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:29:42.359373 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356081 4471 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:29:42.359373 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356084 4471 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:29:42.359373 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356086 4471 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:29:42.359373 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356089 4471 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:29:42.359373 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356091 4471 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:29:42.359373 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356094 4471 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:29:42.359373 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356096 4471 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:29:42.359373 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356101 4471 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:29:42.359373 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356104 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:29:42.359373 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356106 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:29:42.359373 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356109 4471 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:29:42.359861 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356112 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:29:42.359861 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356115 4471 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:29:42.359861 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356119 4471 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:29:42.359861 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356121 4471 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:29:42.359861 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356124 4471 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:29:42.359861 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356127 4471 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:29:42.359861 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356129 4471 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:29:42.359861 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356132 4471 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:29:42.359861 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356134 4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:29:42.359861 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356137 4471 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:29:42.359861 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356139 4471 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:29:42.359861 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356142 4471 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:29:42.359861 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356144 4471 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:29:42.359861 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356147 4471 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:29:42.359861 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356659 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:29:42.359861 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356666 4471 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:29:42.359861 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356670 4471 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:29:42.359861 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356674 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:29:42.359861 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356677 4471 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:29:42.359861 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356680 4471 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:29:42.360367 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356683 4471 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:29:42.360367 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356686 4471 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:29:42.360367 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356689 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:29:42.360367 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356691 4471 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:29:42.360367 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356694 4471 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:29:42.360367 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356696 4471 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:29:42.360367 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356699 4471 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:29:42.360367 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356702 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:29:42.360367 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356706 4471 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:29:42.360367 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356709 4471 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:29:42.360367 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356711 4471 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:29:42.360367 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356714 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:29:42.360367 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356718 4471 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:29:42.360367 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356721 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:29:42.360367 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356724 4471 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:29:42.360367 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356726 4471 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:29:42.360367 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356729 4471 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:29:42.360367 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356731 4471 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:29:42.360367 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356734 4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:29:42.360367 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356736 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:29:42.360981 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356739 4471 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:29:42.360981 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356743 4471 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:29:42.360981 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356747 4471 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:29:42.360981 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356750 4471 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:29:42.360981 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356753 4471 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:29:42.360981 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356756 4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:29:42.360981 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356758 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:29:42.360981 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356761 4471 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:29:42.360981 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356763 4471 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:29:42.360981 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356766 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:29:42.360981 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356769 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:29:42.360981 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356772 4471 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:29:42.360981 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356774 4471 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:29:42.360981 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356776 4471 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:29:42.360981 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356779 4471 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:29:42.360981 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356782 4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:29:42.360981 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356784 4471 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:29:42.360981 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356787 4471 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:29:42.360981 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356790 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:29:42.361787 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356792 4471 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:29:42.361787 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356796 4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:29:42.361787 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356798 4471 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:29:42.361787 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356801 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:29:42.361787 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356803 4471 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:29:42.361787 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356810 4471 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:29:42.361787 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356813 4471 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:29:42.361787 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356815 4471 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:29:42.361787 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356818 4471 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:29:42.361787 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356820 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:29:42.361787 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356823 4471 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:29:42.361787 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356826 4471 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:29:42.361787 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356828 4471 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:29:42.361787 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356831 4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:29:42.361787 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356833 4471 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:29:42.361787 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356836 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:29:42.361787 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356838 4471 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:29:42.361787 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356841 4471 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:29:42.361787 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356844 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:29:42.361787 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356846 4471 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:29:42.362299 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356849 4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:29:42.362299 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356851 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:29:42.362299 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356854 4471 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:29:42.362299 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356857 4471 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:29:42.362299 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356860 4471 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:29:42.362299 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356862 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:29:42.362299 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356865 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:29:42.362299 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356868 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:29:42.362299 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356870 4471 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:29:42.362299 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356873 4471 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:29:42.362299 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356875 4471 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:29:42.362299 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356878 4471 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:29:42.362299 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356881 4471 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:29:42.362299 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356886 4471 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:29:42.362299 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356889 4471 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:29:42.362299 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356891 4471 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:29:42.362299 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356894 4471 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:29:42.362299 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356898 4471 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:29:42.362299 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356901 4471 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:29:42.362299 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356904 4471 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:29:42.362823 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356906 4471 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:29:42.362823 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356909 4471 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:29:42.362823 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356912 4471 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:29:42.362823 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356914 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:29:42.362823 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356917 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:29:42.362823 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356919 4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:29:42.362823 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356922 4471 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:29:42.362823 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356925 4471 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:29:42.362823 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356927 4471 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:29:42.362823 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356930 4471 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:29:42.362823 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356932 4471 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:29:42.362823 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356935 4471 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:29:42.362823 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356938 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:29:42.362823 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356941 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:29:42.362823 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356943 4471 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:29:42.362823 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356946 4471 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:29:42.362823 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356948 4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:29:42.362823 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356951 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:29:42.362823 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356953 4471 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:29:42.363294 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356956 4471 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:29:42.363294 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.356958 4471 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:29:42.363294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357057 4471 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 17:29:42.363294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357067 4471 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 17:29:42.363294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357076 4471 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 17:29:42.363294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357083 4471 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 17:29:42.363294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357088 4471 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 17:29:42.363294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357092 4471 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 17:29:42.363294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357096 4471 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 17:29:42.363294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357100 4471 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 17:29:42.363294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357105 4471 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 17:29:42.363294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357108 4471 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 17:29:42.363294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357113 4471 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 17:29:42.363294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357116 4471 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 17:29:42.363294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357119 4471 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 17:29:42.363294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357123 4471 flags.go:64] FLAG: --cgroup-root="" Nov 02 17:29:42.363294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357125 4471 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 17:29:42.363294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357128 4471 flags.go:64] FLAG: --client-ca-file="" Nov 02 17:29:42.363294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357131 4471 flags.go:64] FLAG: --cloud-provider="external" Nov 02 17:29:42.363294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357134 4471 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 17:29:42.363294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357138 4471 flags.go:64] FLAG: --cluster-domain="" Nov 02 17:29:42.363294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357140 4471 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 17:29:42.363294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357143 4471 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 17:29:42.363294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357147 4471 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 17:29:42.363918 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357150 4471 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 17:29:42.363918 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357154 4471 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 17:29:42.363918 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357157 4471 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 17:29:42.363918 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357160 4471 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 17:29:42.363918 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357164 4471 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 17:29:42.363918 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357167 4471 flags.go:64] FLAG: --contention-profiling="false" Nov 02 17:29:42.363918 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357170 4471 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 17:29:42.363918 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357173 4471 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 17:29:42.363918 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357176 4471 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 17:29:42.363918 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357179 4471 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 17:29:42.363918 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357183 4471 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 17:29:42.363918 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357186 4471 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 17:29:42.363918 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357189 4471 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 17:29:42.363918 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357192 4471 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 17:29:42.363918 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357196 4471 flags.go:64] FLAG: --enable-server="true" Nov 02 17:29:42.363918 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357199 4471 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 17:29:42.363918 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357203 4471 flags.go:64] FLAG: --event-burst="100" Nov 02 17:29:42.363918 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357206 4471 flags.go:64] FLAG: --event-qps="50" Nov 02 17:29:42.363918 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357211 4471 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 17:29:42.363918 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357214 4471 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 17:29:42.363918 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357217 4471 flags.go:64] FLAG: --eviction-hard="" Nov 02 17:29:42.363918 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357221 4471 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 17:29:42.363918 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357225 4471 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 17:29:42.363918 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357228 4471 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 17:29:42.363918 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357231 4471 flags.go:64] FLAG: --eviction-soft="" Nov 02 17:29:42.364553 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357234 4471 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 17:29:42.364553 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357237 4471 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 17:29:42.364553 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357240 4471 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 17:29:42.364553 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357243 4471 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 17:29:42.364553 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357246 4471 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 17:29:42.364553 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357249 4471 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 17:29:42.364553 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357252 4471 flags.go:64] FLAG: --feature-gates="" Nov 02 17:29:42.364553 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357256 4471 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 17:29:42.364553 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357259 4471 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 17:29:42.364553 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357263 4471 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 17:29:42.364553 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357266 4471 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 17:29:42.364553 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357269 4471 flags.go:64] FLAG: --healthz-port="10248" Nov 02 17:29:42.364553 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357272 4471 flags.go:64] FLAG: --help="false" Nov 02 17:29:42.364553 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357275 4471 flags.go:64] FLAG: --hostname-override="ip-10-0-142-26.ec2.internal" Nov 02 17:29:42.364553 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357278 4471 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 17:29:42.364553 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357281 4471 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 17:29:42.364553 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357284 4471 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 17:29:42.364553 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357288 4471 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 17:29:42.364553 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357292 4471 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 17:29:42.364553 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357295 4471 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 17:29:42.364553 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357298 4471 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 17:29:42.364553 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357312 4471 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 17:29:42.364553 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357315 4471 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 17:29:42.365223 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357318 4471 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 17:29:42.365223 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357322 4471 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 17:29:42.365223 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357326 4471 flags.go:64] FLAG: --kube-reserved="" Nov 02 17:29:42.365223 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357329 4471 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 17:29:42.365223 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357332 4471 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 17:29:42.365223 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357335 4471 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 17:29:42.365223 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357339 4471 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 17:29:42.365223 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357342 4471 flags.go:64] FLAG: --lock-file="" Nov 02 17:29:42.365223 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357345 4471 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 17:29:42.365223 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357348 4471 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 17:29:42.365223 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357351 4471 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 17:29:42.365223 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357356 4471 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 17:29:42.365223 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357359 4471 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 17:29:42.365223 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357362 4471 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 17:29:42.365223 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357365 4471 flags.go:64] FLAG: --logging-format="text" Nov 02 17:29:42.365223 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357368 4471 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 17:29:42.365223 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357372 4471 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 17:29:42.365223 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357375 4471 flags.go:64] FLAG: --manifest-url="" Nov 02 17:29:42.365223 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357378 4471 flags.go:64] FLAG: --manifest-url-header="" Nov 02 17:29:42.365223 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357383 4471 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 17:29:42.365223 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357386 4471 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 17:29:42.365223 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357390 4471 flags.go:64] FLAG: --max-pods="110" Nov 02 17:29:42.365223 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357393 4471 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 17:29:42.365223 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357396 4471 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 17:29:42.365223 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357399 4471 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 17:29:42.365849 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357402 4471 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 17:29:42.365849 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357405 4471 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 17:29:42.365849 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357408 4471 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 17:29:42.365849 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357411 4471 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 17:29:42.365849 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357418 4471 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 17:29:42.365849 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357421 4471 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 17:29:42.365849 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357424 4471 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 17:29:42.365849 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357428 4471 flags.go:64] FLAG: --pod-cidr="" Nov 02 17:29:42.365849 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357430 4471 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 17:29:42.365849 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357434 4471 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 17:29:42.365849 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357437 4471 flags.go:64] FLAG: --pods-per-core="0" Nov 02 17:29:42.365849 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357440 4471 flags.go:64] FLAG: --port="10250" Nov 02 17:29:42.365849 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357443 4471 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 17:29:42.365849 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357446 4471 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0de57b7ca78057d31" Nov 02 17:29:42.365849 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357450 4471 flags.go:64] FLAG: --qos-reserved="" Nov 02 17:29:42.365849 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357453 4471 flags.go:64] FLAG: --read-only-port="10255" Nov 02 17:29:42.365849 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357456 4471 flags.go:64] FLAG: --register-node="true" Nov 02 17:29:42.365849 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357459 4471 flags.go:64] FLAG: --register-with-taints="" Nov 02 17:29:42.365849 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357462 4471 flags.go:64] FLAG: --registry-burst="10" Nov 02 17:29:42.365849 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357465 4471 flags.go:64] FLAG: --registry-qps="5" Nov 02 17:29:42.365849 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357468 4471 flags.go:64] FLAG: --reserved-cpus="" Nov 02 17:29:42.365849 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357471 4471 flags.go:64] FLAG: --reserved-memory="" Nov 02 17:29:42.365849 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357477 4471 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 17:29:42.365849 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357480 4471 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 17:29:42.365849 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357483 4471 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 17:29:42.365849 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357485 4471 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 17:29:42.366560 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357488 4471 flags.go:64] FLAG: --runonce="false" Nov 02 17:29:42.366560 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357491 4471 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 17:29:42.366560 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357495 4471 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 17:29:42.366560 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357498 4471 flags.go:64] FLAG: --seccomp-default="false" Nov 02 17:29:42.366560 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357500 4471 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 17:29:42.366560 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357503 4471 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 17:29:42.366560 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357506 4471 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 17:29:42.366560 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357509 4471 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 17:29:42.366560 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357512 4471 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 17:29:42.366560 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357515 4471 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 17:29:42.366560 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357517 4471 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 17:29:42.366560 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357521 4471 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 17:29:42.366560 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357523 4471 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 17:29:42.366560 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357526 4471 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 17:29:42.366560 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357529 4471 flags.go:64] FLAG: --system-cgroups="" Nov 02 17:29:42.366560 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357532 4471 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 17:29:42.366560 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357537 4471 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 17:29:42.366560 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357540 4471 flags.go:64] FLAG: --tls-cert-file="" Nov 02 17:29:42.366560 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357543 4471 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 17:29:42.366560 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357546 4471 flags.go:64] FLAG: --tls-min-version="" Nov 02 17:29:42.366560 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357549 4471 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 17:29:42.366560 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357552 4471 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 17:29:42.366560 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357555 4471 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 17:29:42.366560 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357558 4471 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 17:29:42.366560 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357561 4471 flags.go:64] FLAG: --v="2" Nov 02 17:29:42.367205 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357565 4471 flags.go:64] FLAG: --version="false" Nov 02 17:29:42.367205 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357569 4471 flags.go:64] FLAG: --vmodule="" Nov 02 17:29:42.367205 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357573 4471 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 17:29:42.367205 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.357578 4471 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 17:29:42.367205 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357717 4471 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:29:42.367205 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357720 4471 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:29:42.367205 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357725 4471 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:29:42.367205 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357728 4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:29:42.367205 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357730 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:29:42.367205 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357733 4471 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:29:42.367205 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357736 4471 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:29:42.367205 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357738 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:29:42.367205 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357741 4471 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:29:42.367205 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357744 4471 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:29:42.367205 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357746 4471 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:29:42.367205 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357749 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:29:42.367205 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357752 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:29:42.367205 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357754 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:29:42.367205 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357757 4471 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:29:42.367205 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357760 4471 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:29:42.367205 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357762 4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:29:42.367897 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357765 4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:29:42.367897 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357767 4471 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:29:42.367897 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357770 4471 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:29:42.367897 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357772 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:29:42.367897 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357775 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:29:42.367897 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357777 4471 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:29:42.367897 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357780 4471 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:29:42.367897 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357783 4471 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:29:42.367897 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357786 4471 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:29:42.367897 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357788 4471 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:29:42.367897 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357791 4471 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:29:42.367897 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357795 4471 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:29:42.367897 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357799 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:29:42.367897 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357802 4471 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:29:42.367897 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357806 4471 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:29:42.367897 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357809 4471 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:29:42.367897 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357812 4471 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:29:42.367897 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357814 4471 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:29:42.367897 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357817 4471 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:29:42.367897 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357820 4471 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:29:42.368752 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357822 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:29:42.368752 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357825 4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:29:42.368752 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357827 4471 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:29:42.368752 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357830 4471 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:29:42.368752 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357832 4471 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:29:42.368752 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357835 4471 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:29:42.368752 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357837 4471 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:29:42.368752 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357840 4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:29:42.368752 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357842 4471 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:29:42.368752 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357845 4471 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:29:42.368752 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357847 4471 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:29:42.368752 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357850 4471 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:29:42.368752 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357852 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:29:42.368752 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357855 4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:29:42.368752 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357857 4471 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:29:42.368752 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357860 4471 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:29:42.368752 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357862 4471 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:29:42.368752 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357865 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:29:42.368752 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357867 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:29:42.368752 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357872 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:29:42.369462 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357875 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:29:42.369462 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357878 4471 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:29:42.369462 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357881 4471 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:29:42.369462 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357883 4471 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:29:42.369462 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357886 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:29:42.369462 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357888 4471 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:29:42.369462 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357892 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:29:42.369462 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357895 4471 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:29:42.369462 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357898 4471 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:29:42.369462 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357901 4471 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:29:42.369462 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357903 4471 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:29:42.369462 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357906 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:29:42.369462 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357908 4471 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:29:42.369462 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357911 4471 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:29:42.369462 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357913 4471 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:29:42.369462 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357916 4471 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:29:42.369462 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357919 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:29:42.369462 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357921 4471 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:29:42.369462 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357924 4471 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:29:42.370174 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357927 4471 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:29:42.370174 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357929 4471 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:29:42.370174 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357932 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:29:42.370174 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357935 4471 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:29:42.370174 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357937 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:29:42.370174 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357940 4471 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:29:42.370174 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357943 4471 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:29:42.370174 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357945 4471 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:29:42.370174 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357948 4471 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:29:42.370174 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357951 4471 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:29:42.370174 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357953 4471 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:29:42.370174 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357956 4471 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:29:42.370174 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357961 4471 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:29:42.370174 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357965 4471 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:29:42.370174 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357969 4471 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:29:42.370174 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357972 4471 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:29:42.370174 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357975 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:29:42.370174 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357978 4471 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:29:42.370174 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357980 4471 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:29:42.370824 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357984 4471 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:29:42.370824 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357986 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:29:42.370824 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357989 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:29:42.370824 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357992 4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:29:42.370824 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357994 4471 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:29:42.370824 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.357997 4471 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:29:42.370824 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358000 4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:29:42.370824 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358002 4471 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:29:42.370824 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358005 4471 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:29:42.370824 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358007 4471 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:29:42.370824 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358010 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:29:42.370824 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.358037 4471 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:29:42.370824 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358079 4471 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:29:42.370824 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358084 4471 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:29:42.370824 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358089 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:29:42.371324 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358092 4471 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:29:42.371324 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358095 4471 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:29:42.371324 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358097 4471 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:29:42.371324 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358100 4471 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:29:42.371324 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358103 4471 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:29:42.371324 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358105 4471 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:29:42.371324 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358108 4471 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:29:42.371324 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358110 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:29:42.371324 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358113 4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:29:42.371324 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358116 4471 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:29:42.371324 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358118 4471 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:29:42.371324 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358123 4471 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:29:42.371324 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358125 4471 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:29:42.371324 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358128 4471 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:29:42.371324 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358131 4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:29:42.371324 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358134 4471 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:29:42.371324 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358136 4471 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:29:42.371324 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358140 4471 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:29:42.371324 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358143 4471 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:29:42.371324 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358145 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:29:42.371879 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358148 4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:29:42.371879 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358150 4471 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:29:42.371879 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358153 4471 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:29:42.371879 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358155 4471 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:29:42.371879 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358159 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:29:42.371879 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358161 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:29:42.371879 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358164 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:29:42.371879 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358167 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:29:42.371879 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358169 4471 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:29:42.371879 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358172 4471 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:29:42.371879 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358174 4471 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:29:42.371879 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358177 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:29:42.371879 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358179 4471 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:29:42.371879 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358182 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:29:42.371879 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358185 4471 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:29:42.371879 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358187 4471 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:29:42.371879 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358190 4471 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:29:42.371879 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358193 4471 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:29:42.371879 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358195 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:29:42.372408 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358198 4471 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:29:42.372408 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358200 4471 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:29:42.372408 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358202 4471 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:29:42.372408 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358205 4471 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:29:42.372408 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358209 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:29:42.372408 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358211 4471 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:29:42.372408 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358214 4471 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:29:42.372408 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358216 4471 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:29:42.372408 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358220 4471 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:29:42.372408 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358222 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:29:42.372408 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358226 4471 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:29:42.372408 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358228 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:29:42.372408 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358231 4471 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:29:42.372408 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358234 4471 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:29:42.372408 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358236 4471 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:29:42.372408 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358239 4471 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:29:42.372408 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358242 4471 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:29:42.372408 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358244 4471 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:29:42.372408 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358247 4471 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:29:42.372880 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358252 4471 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:29:42.372880 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358255 4471 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:29:42.372880 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358258 4471 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:29:42.372880 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358261 4471 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:29:42.372880 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358264 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:29:42.372880 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358267 4471 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:29:42.372880 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358269 4471 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:29:42.372880 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358272 4471 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:29:42.372880 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358274 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:29:42.372880 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358277 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:29:42.372880 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358280 4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:29:42.372880 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358282 4471 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:29:42.372880 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358285 4471 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:29:42.372880 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358287 4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:29:42.372880 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358290 4471 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:29:42.372880 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358293 4471 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:29:42.372880 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358295 4471 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:29:42.372880 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358299 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:29:42.372880 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358312 4471 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:29:42.373356 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358315 4471 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:29:42.373356 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358318 4471 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:29:42.373356 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358321 4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:29:42.373356 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358324 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:29:42.373356 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358328 4471 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:29:42.373356 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358331 4471 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:29:42.373356 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358333 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:29:42.373356 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358336 4471 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:29:42.373356 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358338 4471 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:29:42.373356 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358341 4471 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:29:42.373356 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358344 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:29:42.373356 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358347 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:29:42.373356 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358349 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:29:42.373356 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358352 4471 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:29:42.373356 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358354 4471 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:29:42.373356 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358357 4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:29:42.373356 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358360 4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:29:42.373356 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358362 4471 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:29:42.373356 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358365 4471 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:29:42.373356 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358367 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:29:42.373864 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358370 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:29:42.373864 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358374 4471 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:29:42.373864 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358376 4471 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:29:42.373864 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358379 4471 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:29:42.373864 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358382 4471 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:29:42.373864 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.358384 4471 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:29:42.373864 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.363217 4471 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 17:29:42.373864 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.363234 4471 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 17:29:42.373864 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.363253 4471 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 17:29:42.373864 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.363258 4471 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 17:29:42.373864 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363334 4471 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:29:42.373864 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363340 4471 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:29:42.373864 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363343 4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:29:42.373864 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363346 4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:29:42.373864 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363349 4471 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:29:42.373864 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363352 4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:29:42.373864 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363355 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:29:42.373864 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363358 4471 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:29:42.373864 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363362 4471 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:29:42.374855 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363366 4471 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:29:42.374855 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363370 4471 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:29:42.374855 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363372 4471 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:29:42.374855 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363375 4471 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:29:42.374855 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363378 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:29:42.374855 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363381 4471 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:29:42.374855 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363383 4471 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:29:42.374855 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363386 4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:29:42.374855 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363389 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:29:42.374855 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363392 4471 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:29:42.374855 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363394 4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:29:42.374855 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363397 4471 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:29:42.374855 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363400 4471 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:29:42.374855 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363403 4471 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:29:42.374855 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363406 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:29:42.374855 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363409 4471 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:29:42.374855 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363412 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:29:42.374855 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363420 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:29:42.374855 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363423 4471 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:29:42.374855 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363426 4471 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:29:42.375536 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363428 4471 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:29:42.375536 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363431 4471 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:29:42.375536 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363434 4471 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:29:42.375536 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363437 4471 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:29:42.375536 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363439 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:29:42.375536 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363442 4471 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:29:42.375536 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363445 4471 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:29:42.375536 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363448 4471 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:29:42.375536 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363450 4471 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:29:42.375536 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363453 4471 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:29:42.375536 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363456 4471 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:29:42.375536 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363459 4471 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:29:42.375536 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363461 4471 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:29:42.375536 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363464 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:29:42.375536 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363467 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:29:42.375536 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363470 4471 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:29:42.375536 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363473 4471 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:29:42.375536 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363484 4471 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:29:42.375536 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363487 4471 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:29:42.375536 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363490 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:29:42.376054 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363492 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:29:42.376054 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363495 4471 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:29:42.376054 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363497 4471 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:29:42.376054 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363500 4471 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:29:42.376054 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363503 4471 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:29:42.376054 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363506 4471 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:29:42.376054 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363509 4471 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:29:42.376054 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363512 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:29:42.376054 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363514 4471 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:29:42.376054 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363517 4471 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:29:42.376054 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363520 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:29:42.376054 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363522 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:29:42.376054 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363525 4471 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:29:42.376054 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363527 4471 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:29:42.376054 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363530 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:29:42.376054 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363533 4471 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:29:42.376054 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363535 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:29:42.376054 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363538 4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:29:42.376054 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363540 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:29:42.376054 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363543 4471 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:29:42.376584 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363546 4471 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:29:42.376584 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363549 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:29:42.376584 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363552 4471 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:29:42.376584 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363554 4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:29:42.376584 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363557 4471 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:29:42.376584 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363560 4471 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:29:42.376584 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363562 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:29:42.376584 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363565 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:29:42.376584 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363568 4471 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:29:42.376584 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363570 4471 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:29:42.376584 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363577 4471 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:29:42.376584 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363580 4471 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:29:42.376584 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363582 4471 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:29:42.376584 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363585 4471 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:29:42.376584 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363588 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:29:42.376584 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363591 4471 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:29:42.376584 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363594 4471 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:29:42.376584 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363597 4471 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:29:42.376584 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363602 4471 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:29:42.377065 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363606 4471 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:29:42.377065 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363609 4471 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:29:42.377065 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363612 4471 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:29:42.377065 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363615 4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:29:42.377065 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363619 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:29:42.377065 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363621 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:29:42.377065 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363624 4471 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:29:42.377065 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363627 4471 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:29:42.377065 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363630 4471 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:29:42.377065 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363632 4471 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:29:42.377065 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363635 4471 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:29:42.377065 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363638 4471 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:29:42.377065 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363641 4471 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:29:42.377065 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363643 4471 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:29:42.377065 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363646 4471 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:29:42.377065 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363649 4471 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:29:42.377065 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363652 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:29:42.377065 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363654 4471 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:29:42.377527 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.363683 4471 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:29:42.377527 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363827 4471 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:29:42.377527 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363831 4471 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:29:42.377527 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363834 4471 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:29:42.377527 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363837 4471 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:29:42.377527 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363839 4471 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:29:42.377527 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363843 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:29:42.377527 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363846 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:29:42.377527 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363849 4471 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:29:42.377527 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363851 4471 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:29:42.377527 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363854 4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:29:42.377527 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363857 4471 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:29:42.377527 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363860 4471 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:29:42.377527 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363863 4471 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:29:42.377527 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363867 4471 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:29:42.377898 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363870 4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:29:42.377898 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363873 4471 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:29:42.377898 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363876 4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:29:42.377898 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363878 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:29:42.377898 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363881 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:29:42.377898 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363884 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:29:42.377898 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363886 4471 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:29:42.377898 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363889 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:29:42.377898 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363891 4471 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:29:42.377898 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363894 4471 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:29:42.377898 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363896 4471 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:29:42.377898 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363899 4471 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:29:42.377898 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363902 4471 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:29:42.377898 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363904 4471 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:29:42.377898 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363907 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:29:42.377898 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363910 4471 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:29:42.377898 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363913 4471 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:29:42.377898 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363915 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:29:42.377898 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363918 4471 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:29:42.378437 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363920 4471 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:29:42.378437 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363923 4471 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:29:42.378437 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363926 4471 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:29:42.378437 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363928 4471 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:29:42.378437 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363931 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:29:42.378437 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363934 4471 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:29:42.378437 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363937 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:29:42.378437 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363940 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:29:42.378437 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363943 4471 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:29:42.378437 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363946 4471 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:29:42.378437 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363948 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:29:42.378437 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363951 4471 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:29:42.378437 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363953 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:29:42.378437 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363956 4471 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:29:42.378437 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363958 4471 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:29:42.378437 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363961 4471 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:29:42.378437 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363963 4471 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:29:42.378437 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363966 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:29:42.378437 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363968 4471 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:29:42.378437 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363971 4471 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:29:42.378920 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363974 4471 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:29:42.378920 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363976 4471 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:29:42.378920 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363979 4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:29:42.378920 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363982 4471 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:29:42.378920 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363984 4471 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:29:42.378920 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363987 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:29:42.378920 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363989 4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:29:42.378920 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363992 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:29:42.378920 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363995 4471 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:29:42.378920 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.363997 4471 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:29:42.378920 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364000 4471 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:29:42.378920 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364003 4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:29:42.378920 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364005 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:29:42.378920 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364008 4471 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:29:42.378920 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364010 4471 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:29:42.378920 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364013 4471 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:29:42.378920 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364015 4471 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:29:42.378920 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364018 4471 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:29:42.378920 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364021 4471 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:29:42.378920 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364024 4471 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:29:42.379418 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364027 4471 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:29:42.379418 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364029 4471 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:29:42.379418 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364031 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:29:42.379418 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364034 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:29:42.379418 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364036 4471 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:29:42.379418 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364039 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:29:42.379418 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364041 4471 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:29:42.379418 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364044 4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:29:42.379418 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364046 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:29:42.379418 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364049 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:29:42.379418 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364051 4471 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:29:42.379418 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364054 4471 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:29:42.379418 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364058 4471 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:29:42.379418 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364061 4471 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:29:42.379418 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364065 4471 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:29:42.379418 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364067 4471 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:29:42.379418 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364070 4471 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:29:42.379418 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364073 4471 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:29:42.379418 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364076 4471 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:29:42.379869 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364079 4471 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:29:42.379869 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364081 4471 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:29:42.379869 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364084 4471 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:29:42.379869 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364087 4471 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:29:42.379869 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364089 4471 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:29:42.379869 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364092 4471 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:29:42.379869 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364094 4471 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:29:42.379869 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364097 4471 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:29:42.379869 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364099 4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:29:42.379869 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364102 4471 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:29:42.379869 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364104 4471 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:29:42.379869 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364107 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:29:42.379869 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364110 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:29:42.379869 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364113 4471 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:29:42.379869 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.364137 4471 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:29:42.379869 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364168 4471 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:29:42.380275 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364172 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:29:42.380275 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364175 4471 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:29:42.380275 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364178 4471 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:29:42.380275 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364181 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:29:42.380275 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364183 4471 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:29:42.380275 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364186 4471 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:29:42.380275 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364188 4471 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:29:42.380275 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364191 4471 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:29:42.380275 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364193 4471 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:29:42.380275 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364196 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:29:42.380275 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364199 4471 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:29:42.380275 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364204 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:29:42.380275 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364206 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:29:42.380275 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364209 4471 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:29:42.380275 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364212 4471 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:29:42.380275 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364215 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:29:42.380275 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364217 4471 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:29:42.380275 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364220 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:29:42.380275 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364222 4471 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:29:42.380275 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364225 4471 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:29:42.380803 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364228 4471 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:29:42.380803 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364230 4471 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:29:42.380803 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364242 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:29:42.380803 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364245 4471 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:29:42.380803 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364247 4471 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:29:42.380803 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364250 4471 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:29:42.380803 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364253 4471 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:29:42.380803 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364255 4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:29:42.380803 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364258 4471 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:29:42.380803 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364261 4471 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:29:42.380803 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364264 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:29:42.380803 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364267 4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:29:42.380803 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364269 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:29:42.380803 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364272 4471 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:29:42.380803 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364275 4471 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:29:42.380803 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364277 4471 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:29:42.380803 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364280 4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:29:42.380803 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364283 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:29:42.380803 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364285 4471 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:29:42.380803 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364288 4471 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:29:42.381396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364290 4471 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:29:42.381396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364293 4471 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:29:42.381396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364295 4471 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:29:42.381396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364300 4471 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:29:42.381396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364320 4471 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:29:42.381396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364324 4471 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:29:42.381396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364329 4471 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:29:42.381396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364350 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:29:42.381396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364355 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:29:42.381396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364358 4471 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:29:42.381396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364360 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:29:42.381396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364364 4471 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:29:42.381396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364366 4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:29:42.381396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364369 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:29:42.381396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364371 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:29:42.381396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364374 4471 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:29:42.381396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364377 4471 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:29:42.381396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364381 4471 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:29:42.381396 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364384 4471 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:29:42.381912 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364387 4471 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:29:42.381912 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364390 4471 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:29:42.381912 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364392 4471 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:29:42.381912 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364396 4471 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:29:42.381912 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364398 4471 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:29:42.381912 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364401 4471 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:29:42.381912 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364404 4471 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:29:42.381912 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364407 4471 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:29:42.381912 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364410 4471 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:29:42.381912 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364413 4471 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:29:42.381912 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364415 4471 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:29:42.381912 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364417 4471 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:29:42.381912 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364420 4471 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:29:42.381912 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364423 4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:29:42.381912 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364425 4471 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:29:42.381912 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364428 4471 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:29:42.381912 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364432 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:29:42.381912 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364435 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:29:42.381912 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364437 4471 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:29:42.381912 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364440 4471 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:29:42.382693 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364443 4471 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:29:42.382693 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364445 4471 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:29:42.382693 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364448 4471 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:29:42.382693 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364451 4471 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:29:42.382693 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364453 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:29:42.382693 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364456 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:29:42.382693 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364459 4471 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:29:42.382693 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364461 4471 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:29:42.382693 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364464 4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:29:42.382693 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364466 4471 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:29:42.382693 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364469 4471 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:29:42.382693 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364471 4471 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:29:42.382693 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364475 4471 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:29:42.382693 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364478 4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:29:42.382693 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364481 4471 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:29:42.382693 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364484 4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:29:42.382693 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364487 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:29:42.382693 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364489 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:29:42.382693 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364492 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:29:42.383239 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364494 4471 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:29:42.383239 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364497 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:29:42.383239 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364499 4471 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:29:42.383239 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364502 4471 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:29:42.383239 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364505 4471 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:29:42.383239 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364509 4471 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:29:42.383239 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:42.364511 4471 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:29:42.383239 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.364568 4471 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 17:29:42.383239 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.365561 4471 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 17:29:42.383239 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.365615 4471 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 17:29:42.383239 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.365790 4471 server.go:1014] "Starting client certificate rotation" Nov 02 17:29:42.383239 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.365944 4471 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 17:29:42.383239 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.366010 4471 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 17:24:02 +0000 UTC" deadline="2028-07-02 04:11:40.977507829 +0000 UTC" Nov 02 17:29:42.383239 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.366073 4471 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14578h41m58.611436597s" Nov 02 17:29:42.383239 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.366254 4471 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 17:29:42.383239 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.367382 4471 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 17:29:42.383660 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.369100 4471 log.go:25] "Validated CRI v1 runtime API" Nov 02 17:29:42.383660 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.371013 4471 log.go:25] "Validated CRI v1 image API" Nov 02 17:29:42.383660 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.372010 4471 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 17:29:42.383660 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.373457 4471 fs.go:135] Filesystem UUIDs: map[20b21f6f-361d-4dee-8cd2-0fa25a362b67:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 86cc3286-62a7-4e7b-a5eb-d90d3e1f0913:/dev/nvme0n1p3] Nov 02 17:29:42.383815 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.373484 4471 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1857c4a0aa8bf1c3921d4ad577b26ade06fc0d3309705f5aebabf1cce95a1b1b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1857c4a0aa8bf1c3921d4ad577b26ade06fc0d3309705f5aebabf1cce95a1b1b/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/233c9f1f93bfdf7c1b0ed3b4e671557d2b6d78d24d586f3c0cd7ea694c5ab72d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/233c9f1f93bfdf7c1b0ed3b4e671557d2b6d78d24d586f3c0cd7ea694c5ab72d/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32cb396d6ecfbbd76e588b0385a3c086a521faf4491e3bb2b2130e7ac232ea85/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32cb396d6ecfbbd76e588b0385a3c086a521faf4491e3bb2b2130e7ac232ea85/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4037f20116599db676be2e3508ddaab64598b45fdcd9bfabeef17b59157040bc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4037f20116599db676be2e3508ddaab64598b45fdcd9bfabeef17b59157040bc/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/42ca45cc6abd1e597a513f448d8d64c0af835aaaa024e1559332ae203fb2799a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/42ca45cc6abd1e597a513f448d8d64c0af835aaaa024e1559332ae203fb2799a/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7fa25d80d54565f9834c98d65add99ab446169d815d3234a2f109707ff8b1893/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7fa25d80d54565f9834c98d65add99ab446169d815d3234a2f109707ff8b1893/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80dcc1078096451d0c8da3583979ee66c7a924f6d74409220f0c7ebee2ad3c46/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80dcc1078096451d0c8da3583979ee66c7a924f6d74409220f0c7ebee2ad3c46/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a67aea3907d74aacfa4e7d623918e1e259725faab454313457660e49c28e754/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a67aea3907d74aacfa4e7d623918e1e259725faab454313457660e49c28e754/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d764223f8703f2c068280733e8df67e4b15cf89bec7e9f059b0baae9c2d2e91/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d764223f8703f2c068280733e8df67e4b15cf89bec7e9f059b0baae9c2d2e91/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a3499a029d1384764477256caca4907bf36720f5824334057f9a4b48f9732ecb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a3499a029d1384764477256caca4907bf36720f5824334057f9a4b48f9732ecb/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aa47ee906c3a9ec04b42805201a94244a95b55047a0d8a219e86cb829b53fae6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aa47ee906c3a9ec04b42805201a94244a95b55047a0d8a219e86cb829b53fae6/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d59d6ed36bf74eafed1a8ed10162ada70e1d312a26ed4f76ddb9c16631588825/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d59d6ed36bf74eafed1a8ed10162ada70e1d312a26ed4f76ddb9c16631588825/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e68c4c0e2265daed54e886e09441c2e9d3230250f828fd57699f5ab28fbcd936/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e68c4c0e2265daed54e886e09441c2e9d3230250f828fd57699f5ab28fbcd936/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e77524a03d85053d8c4fd96128de6a0e0add551fcae0300e84c5f2811506a1e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e77524a03d85053d8c4fd96128de6a0e0add551fcae0300e84c5f2811506a1e3/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/faa537da1489a54bb0d2bfde98e63d93a68abf0c1eb1b471055ada636900af82/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/faa537da1489a54bb0d2bfde98e63d93a68abf0c1eb1b471055ada636900af82/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/007d7841-411c-4b0f-b8aa-d36097b6bf31/volumes/kubernetes.io~projected/kube-api-access-pmjd4:{mountpoint:/var/lib/kubelet/pods/007d7841-411c-4b0f-b8aa-d36097b6bf31/volumes/kubernetes.io~projected/kube-api-access-pmjd4 major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0aed3a0d-f6c0-4955-8d67-8b8b98bebe25/volumes/kubernetes.io~projected/kube-api-access-cg9fq:{mountpoint:/var/lib/kubelet/pods/0aed3a0d-f6c0-4955-8d67-8b8b98bebe25/volumes/kubernetes.io~projected/kube-api-access-cg9fq major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b0e99b5-4469-44ac-b241-a6edb43bc8ec/volumes/kubernetes.io~projected/kube-api-access-zgkcf:{mountpoint:/var/lib/kubelet/pods/0b0e99b5-4469-44ac-b241-a6edb43bc8ec/volumes/kubernetes.io~projected/kube-api-access-zgkcf major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9/volumes/kubernetes.io~projected/kube-api-access-jcgm2:{mountpoint:/var/lib/kubelet/pods/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9/volumes/kubernetes.io~projected/kube-api-access-jcgm2 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b5ee10f-87a1-4168-904e-ba233f304fe3/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/1b5ee10f-87a1-4168-904e-ba233f304fe3/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35ab9b40-d988-40ff-87c0-0cf822cfd863/volumes/kubernetes.io~projected/kube-api-access-4rcc6:{mountpoint:/var/lib/kubelet/pods/35ab9b40-d988-40ff-87c0-0cf822cfd863/volumes/kubernetes.io~projected/kube-api-access-4rcc6 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a4bf99d-1b04-48a4-bca9-dd3f484ad0a8/volumes/kubernetes.io~projected/kube-api-access-4qcnl:{mountpoint:/var/lib/kubelet/pods/3a4bf99d-1b04-48a4-bca9-dd3f484ad0a8/volumes/kubernetes.io~projected/kube-api-access-4qcnl major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d601642-0fa6-4b68-b9cf-324c59e41637/volumes/kubernetes.io~projected/kube-api-access-wvnsx:{mountpoint:/var/lib/kubelet/pods/4d601642-0fa6-4b68-b9cf-324c59e41637/volumes/kubernetes.io~projected/kube-api-access-wvnsx major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5cb013c0-4887-488b-867f-841fb6b01ca5/volumes/kubernetes.io~projected/kube-api-access-dgsrn:{mountpoint:/var/lib/kubelet/pods/5cb013c0-4887-488b-867f-841fb6b01ca5/volumes/kubernetes.io~projected/kube-api-access-dgsrn major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61135006-920f-4fb5-8ba5-c98d6beae6cf/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/61135006-920f-4fb5-8ba5-c98d6beae6cf/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61135006-920f-4fb5-8ba5-c98d6beae6cf/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/61135006-920f-4fb5-8ba5-c98d6beae6cf/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61135006-920f-4fb5-8ba5-c98d6beae6cf/volumes/kubernetes.io~projected/kube-api-access-9fskz:{mountpoint:/var/lib/kubelet/pods/61135006-920f-4fb5-8ba5-c98d6beae6cf/volumes/kubernetes.io~projected/kube-api-access-9fskz major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/665ac753-596e-4d13-b8c1-7a1c7e791cf8/volumes/kubernetes.io~projected/kube-api-access-24tpd:{mountpoint:/var/lib/kubelet/pods/665ac753-596e-4d13-b8c1-7a1c7e791cf8/volumes/kubernetes.io~projected/kube-api-access-24tpd major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81019158-9e01-4dd1-a405-d1cf75d6dc78/volumes/kubernetes.io~projected/kube-api-access-pk945:{mountpoint:/var/lib/kubelet/pods/81019158-9e01-4dd1-a405-d1cf75d6dc78/volumes/kubernetes.io~projected/kube-api-access-pk945 major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81019158-9e01-4dd1-a405-d1cf75d6dc78/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/81019158-9e01-4dd1-a405-d1cf75d6dc78/volumes/kubernetes.io~secret/signing-key major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83c827e3-5338-49b5-97a1-56bdb5904500/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/83c827e3-5338-49b5-97a1-56bdb5904500/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c38cc772-7b2a-4a50-b230-e8b790647ae0/volumes/kubernetes.io~projected/kube-api-access-88l9x:{mountpoint:/var/lib/kubelet/pods/c38cc772-7b2a-4a50-b230-e8b790647ae0/volumes/kubernetes.io~projected/kube-api-access-88l9x major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d09569ff-f9b3-42d4-b163-fae46612bffc/volumes/kubernetes.io~projected/kube-api-access-cv9jt:{mountpoint:/var/lib/kubelet/pods/d09569ff-f9b3-42d4-b163-fae46612bffc/volumes/kubernetes.io~projected/kube-api-access-cv9jt major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d911b628-4ba2-4427-b229-7698d25c2e2c/volumes/kubernetes.io~projected/kube-api-access-dmgzk:{mountpoint:/var/lib/kubelet/pods/d911b628-4ba2-4427-b229-7698d25c2e2c/volumes/kubernetes.io~projected/kube-api-access-dmgzk major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d911b628-4ba2-4427-b229-7698d25c2e2c/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/d911b628-4ba2-4427-b229-7698d25c2e2c/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b/volumes/kubernetes.io~projected/kube-api-access-8b57h:{mountpoint:/var/lib/kubelet/pods/fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b/volumes/kubernetes.io~projected/kube-api-access-8b57h major:0 minor:64 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/cb55f1db02a74d0559dbb991969171ea45652a2d7aef1f8271da39a81aced1c2/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/5705af102646fc5ff1c50f093f862c97d49d21548975aec311ddfc40705becff/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/e01efa271a3c07963287a51306156a3a70a652e67228f7edba7b267da4db7236/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/10101ba7d5c827fd5d91ec90945669c1a023de95684e18a326f055727c89d795/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/ffd4910d2360c1f73b36a6311fc5d2925ab7f4a70d1bd3b5b17d175d9a5bd241/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/c8e56f73cb93a9d33534aff39b783da6f259c67e7f06ceab62486af436087535/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/55d1e9a07a4f63acf4eca2c161b0d1a684b226c594f040edda75c430f53a9516/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/abefce10d4f5089f29c080f2b00dfb6bcd39990b6fbd9e030a65c9c091c11e8c/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/60ba91ce748228ca2fca5123002a8fd826e9b28de46658cfa51fd53e88518a98/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/ab09cb40321b84bba88cb59b6badadbf3518b9154f42ea1d201d9b6b6bb21aa4/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/f54d5fb377a8aa41985ca8177c06a49e1e6f03b851f48e87198caa808001ff1b/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/5fb651f7d527771f6b2f9c53fdf0831603b303f3ad6e1482924356d0c571330b/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/45f6b29022e6a702701a55680a73891744c6fb9846e8fc7c1a806f5a25538c6f/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/c178945018d4b65896526508d22aa070e25af03ac32eed6ad06cee1c1d8006ca/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/4447c118d803ab2acf58210906fbf7629c8c1c73544830e03b01093f24464cd5/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/cc7ae27fb8b69e63b5b35615a0966a308fabaccd85c708491328570248cb2292/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/b8ed9f6db93d307ca8e350b2f5eab50a380703ee50c3983fc0353b4a42e48fb0/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/ea493cea52ace8ceb3e50c0ea97a110c2d9723199c7ff2a47be9791d21949f65/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/9aff0f882bbb001f735eebe151cff9f1ae4c82f864ea21963cf18c717bd5c957/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/36a00a660731ca2797d26a4e65cf02b0b25e708f8869be3fd2b25c2e86b0dd4e/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/49e823fd97c6dda8b6a2c70e1961c7e1789ecabec9e18a3277c1b187f1d612d2/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/cc64b654f5d2a9d7076e25e6ba36532b83ee3d539663d08a85350f485fb87ee9/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-206:{mountpoint:/var/lib/containers/storage/overlay/e8f568c26d7e13d021d270017abb64a9660139ea3e2abdf07915b9bcf5d42b16/merged major:0 minor:206 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/b2ac5b85bda0da4f1b3eb008306796e65340fbeca9b6b012b14d536326751bc2/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/1f676f52e71111c2698089729c42ec9cbf2dbbaf3283cb32358c585194f0d1bd/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/cd4dd92137681f8b8b96ff0354f74bb7248f61004a725f797ed38e0f167a5688/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/76acae3bba2d1a9d86d50665eb55cdc12a7b8ef9e5daf5f9c1841f29e374cdb8/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/6f0bf49854e646f2a5ed6dc5a004945e131d2e1c25186d2233427b9a50200538/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/984cea370ff1dde3981ebd7f205bb48f20795f0020b90ead23da5a66cf623b6e/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/7061a69c42ca052752c402ddda8affc8f084bf08d6fa94f7c5127df5914ca2ef/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/33ef31e8e012e91deda9ddf8e72a0782b5ba102468fce3cbf0a0b0b4417eecad/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/e3d3dd1f74e90f356819b353e30e97b7e06fb1644da659491f50310c7d921e7a/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/76416ab0c645767fa019490378078f3bb2d20d8c0a8d72b841432ed79482011d/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/562bd4634af7faece3e502b3c275c7246bad9ce4ea2c48501fab88dd79bda14f/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/4f8426b45e939576009e6d120899c3f7dfc7488dd70e34c0c1e98c2322d8539d/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/8088eb284de2d7f32c1316686dfab9da2204f45eb20733ab56b36aa0fc8f6104/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/770d51e0862fbccc129fc4b6c501dc4cd92e54c612729ad8899a49267e46a8ac/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/4ed11437e8585ff329e15c08ff54c60f28ba5c70513689d391a787533d971105/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/d258d63467545ac11d56c48719e14575a8a9937c4c7cb7131e49f26dffd6b7bf/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/4787f4c9fdc5891d41ef161da9403ea76f6c3196be321bbf4522ef1e46d9cc9a/merged major:0 minor:98 fsType:overlay blockSize:0}] Nov 02 17:29:42.384015 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.380516 4471 manager.go:217] Machine: {Timestamp:2026-11-02 17:29:42.380044539 +0000 UTC m=+0.075400004 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099890 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec258fbf820ecb9c6938a061d0e49ef5 SystemUUID:ec258fbf-820e-cb9c-6938-a061d0e49ef5 BootID:cc2062ec-e693-427d-8b0c-a65bb170ce11 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/61135006-920f-4fb5-8ba5-c98d6beae6cf/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80dcc1078096451d0c8da3583979ee66c7a924f6d74409220f0c7ebee2ad3c46/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b/volumes/kubernetes.io~projected/kube-api-access-8b57h DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3a4bf99d-1b04-48a4-bca9-dd3f484ad0a8/volumes/kubernetes.io~projected/kube-api-access-4qcnl DeviceMajor:0 DeviceMinor:223 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5cb013c0-4887-488b-867f-841fb6b01ca5/volumes/kubernetes.io~projected/kube-api-access-dgsrn DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/81019158-9e01-4dd1-a405-d1cf75d6dc78/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:228 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/665ac753-596e-4d13-b8c1-7a1c7e791cf8/volumes/kubernetes.io~projected/kube-api-access-24tpd DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0b0e99b5-4469-44ac-b241-a6edb43bc8ec/volumes/kubernetes.io~projected/kube-api-access-zgkcf DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d09569ff-f9b3-42d4-b163-fae46612bffc/volumes/kubernetes.io~projected/kube-api-access-cv9jt DeviceMajor:0 DeviceMinor:213 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1857c4a0aa8bf1c3921d4ad577b26ade06fc0d3309705f5aebabf1cce95a1b1b/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c38cc772-7b2a-4a50-b230-e8b790647ae0/volumes/kubernetes.io~projected/kube-api-access-88l9x DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0aed3a0d-f6c0-4955-8d67-8b8b98bebe25/volumes/kubernetes.io~projected/kube-api-access-cg9fq DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d59d6ed36bf74eafed1a8ed10162ada70e1d312a26ed4f76ddb9c16631588825/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d764223f8703f2c068280733e8df67e4b15cf89bec7e9f059b0baae9c2d2e91/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9/volumes/kubernetes.io~projected/kube-api-access-jcgm2 DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d911b628-4ba2-4427-b229-7698d25c2e2c/volumes/kubernetes.io~projected/kube-api-access-dmgzk DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/233c9f1f93bfdf7c1b0ed3b4e671557d2b6d78d24d586f3c0cd7ea694c5ab72d/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/61135006-920f-4fb5-8ba5-c98d6beae6cf/volumes/kubernetes.io~projected/kube-api-access-9fskz DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a3499a029d1384764477256caca4907bf36720f5824334057f9a4b48f9732ecb/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4037f20116599db676be2e3508ddaab64598b45fdcd9bfabeef17b59157040bc/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-206 DeviceMajor:0 DeviceMinor:206 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/faa537da1489a54bb0d2bfde98e63d93a68abf0c1eb1b471055ada636900af82/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/61135006-920f-4fb5-8ba5-c98d6beae6cf/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d601642-0fa6-4b68-b9cf-324c59e41637/volumes/kubernetes.io~projected/kube-api-access-wvnsx DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7fa25d80d54565f9834c98d65add99ab446169d815d3234a2f109707ff8b1893/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/35ab9b40-d988-40ff-87c0-0cf822cfd863/volumes/kubernetes.io~projected/kube-api-access-4rcc6 DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e77524a03d85053d8c4fd96128de6a0e0add551fcae0300e84c5f2811506a1e3/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aa47ee906c3a9ec04b42805201a94244a95b55047a0d8a219e86cb829b53fae6/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d911b628-4ba2-4427-b229-7698d25c2e2c/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/83c827e3-5338-49b5-97a1-56bdb5904500/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:208 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/007d7841-411c-4b0f-b8aa-d36097b6bf31/volumes/kubernetes.io~projected/kube-api-access-pmjd4 DeviceMajor:0 DeviceMinor:222 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/42ca45cc6abd1e597a513f448d8d64c0af835aaaa024e1559332ae203fb2799a/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32cb396d6ecfbbd76e588b0385a3c086a521faf4491e3bb2b2130e7ac232ea85/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/81019158-9e01-4dd1-a405-d1cf75d6dc78/volumes/kubernetes.io~projected/kube-api-access-pk945 DeviceMajor:0 DeviceMinor:229 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1b5ee10f-87a1-4168-904e-ba233f304fe3/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e68c4c0e2265daed54e886e09441c2e9d3230250f828fd57699f5ab28fbcd936/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a67aea3907d74aacfa4e7d623918e1e259725faab454313457660e49c28e754/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:233c9f1f93bfdf7 MacAddress:6a:70:ff:9a:e9:0b Speed:10000 Mtu:8901} {Name:42ca45cc6abd1e5 MacAddress:a6:14:72:89:61:86 Speed:10000 Mtu:8901} {Name:7fa25d80d54565f MacAddress:a6:85:c0:cd:82:fe Speed:10000 Mtu:8901} {Name:8a67aea3907d74a MacAddress:3a:3f:4f:f4:db:0a Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ec:fe:fa:b3 Speed:0 Mtu:9001} {Name:br-int MacAddress:36:3a:05:ca:77:5e Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:ec:fe:fa:b3 Speed:0 Mtu:9001} {Name:faa537da1489a54 MacAddress:ce:3d:e0:95:45:1f Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:0e:05:41:df:37:33 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:da:86:c7:44:67:49 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 17:29:42.384015 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.380754 4471 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 17:29:42.384015 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.380797 4471 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 17:29:42.384015 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.381129 4471 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 17:29:42.384015 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.381154 4471 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-26.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 17:29:42.384015 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.381295 4471 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 17:29:42.384015 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.381319 4471 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 17:29:42.384015 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.381332 4471 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 17:29:42.384015 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.381349 4471 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 17:29:42.384015 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.381416 4471 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 17:29:42.384015 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.381600 4471 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 17:29:42.384015 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.381710 4471 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 17:29:42.384015 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.381743 4471 kubelet.go:493] "Attempting to sync node with API server" Nov 02 17:29:42.384015 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.381756 4471 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 17:29:42.384015 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.381767 4471 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 17:29:42.384015 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.381776 4471 kubelet.go:405] "Adding apiserver pod source" Nov 02 17:29:42.384015 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.381783 4471 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 17:29:42.384015 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.382683 4471 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 17:29:42.384015 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.382701 4471 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 17:29:42.384015 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.383432 4471 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 17:29:42.392100 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.392083 4471 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 17:29:42.392100 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.392099 4471 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 17:29:42.392203 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.392193 4471 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 17:29:42.392410 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.392398 4471 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 17:29:42.392452 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.392428 4471 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 17:29:42.392518 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.392509 4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 17:29:42.392548 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.392523 4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 17:29:42.392548 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.392529 4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 17:29:42.392548 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.392536 4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 17:29:42.392548 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.392544 4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 17:29:42.392652 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.392553 4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 17:29:42.392652 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.392560 4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 17:29:42.392652 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.392566 4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 17:29:42.392652 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.392572 4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 17:29:42.392652 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.392578 4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 17:29:42.392652 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.392587 4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 17:29:42.392652 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.392598 4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 17:29:42.392652 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.392625 4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 17:29:42.392652 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.392632 4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 17:29:42.395987 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.395975 4471 server.go:1263] "Started kubelet" Nov 02 17:29:42.396117 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.396086 4471 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 17:29:42.396389 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.396078 4471 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 17:29:42.396389 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.396262 4471 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 17:29:42.396524 ip-10-0-142-26 systemd[1]: Started Kubernetes Kubelet. Nov 02 17:29:42.396693 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.396558 4471 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 17:29:42.398418 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.398379 4471 server.go:317] "Adding debug handlers to kubelet server" Nov 02 17:29:42.401036 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.401016 4471 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 17:29:42.401151 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.401134 4471 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 17:29:42.401151 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:29:42.401141 4471 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-26.ec2.internal\" not found" Nov 02 17:29:42.401273 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.401177 4471 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 17:29:42.401273 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.401186 4471 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 17:29:42.401433 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.401412 4471 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 17:29:42.402798 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.401998 4471 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 17:24:04 +0000 UTC" deadline="2028-03-31 14:17:23.737855689 +0000 UTC" Nov 02 17:29:42.402798 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.402043 4471 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12356h47m41.335816738s" Nov 02 17:29:42.403126 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403091 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cb013c0-4887-488b-867f-841fb6b01ca5" volumeName="kubernetes.io/projected/5cb013c0-4887-488b-867f-841fb6b01ca5-kube-api-access-dgsrn" seLinuxMountContext="" Nov 02 17:29:42.403190 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403129 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="665ac753-596e-4d13-b8c1-7a1c7e791cf8" volumeName="kubernetes.io/empty-dir/665ac753-596e-4d13-b8c1-7a1c7e791cf8-tmp-dir" seLinuxMountContext="" Nov 02 17:29:42.403190 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403143 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9" volumeName="kubernetes.io/configmap/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-env-overrides" seLinuxMountContext="" Nov 02 17:29:42.403190 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403154 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c38cc772-7b2a-4a50-b230-e8b790647ae0" volumeName="kubernetes.io/projected/c38cc772-7b2a-4a50-b230-e8b790647ae0-kube-api-access-88l9x" seLinuxMountContext="" Nov 02 17:29:42.403190 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403165 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9" volumeName="kubernetes.io/projected/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-kube-api-access-jcgm2" seLinuxMountContext="" Nov 02 17:29:42.403190 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403176 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35ab9b40-d988-40ff-87c0-0cf822cfd863" volumeName="kubernetes.io/configmap/35ab9b40-d988-40ff-87c0-0cf822cfd863-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 17:29:42.403190 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403188 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61135006-920f-4fb5-8ba5-c98d6beae6cf" volumeName="kubernetes.io/empty-dir/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-tuned" seLinuxMountContext="" Nov 02 17:29:42.403516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403200 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81019158-9e01-4dd1-a405-d1cf75d6dc78" volumeName="kubernetes.io/configmap/81019158-9e01-4dd1-a405-d1cf75d6dc78-signing-cabundle" seLinuxMountContext="" Nov 02 17:29:42.403516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403214 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c38cc772-7b2a-4a50-b230-e8b790647ae0" volumeName="kubernetes.io/configmap/c38cc772-7b2a-4a50-b230-e8b790647ae0-cni-binary-copy" seLinuxMountContext="" Nov 02 17:29:42.403516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403224 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b" volumeName="kubernetes.io/configmap/fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b-iptables-alerter-script" seLinuxMountContext="" Nov 02 17:29:42.403516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403236 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0aed3a0d-f6c0-4955-8d67-8b8b98bebe25" volumeName="kubernetes.io/configmap/0aed3a0d-f6c0-4955-8d67-8b8b98bebe25-serviceca" seLinuxMountContext="" Nov 02 17:29:42.403516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403246 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b5ee10f-87a1-4168-904e-ba233f304fe3" volumeName="kubernetes.io/configmap/1b5ee10f-87a1-4168-904e-ba233f304fe3-konnectivity-ca" seLinuxMountContext="" Nov 02 17:29:42.403516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403260 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35ab9b40-d988-40ff-87c0-0cf822cfd863" volumeName="kubernetes.io/projected/35ab9b40-d988-40ff-87c0-0cf822cfd863-kube-api-access-4rcc6" seLinuxMountContext="" Nov 02 17:29:42.403516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403270 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d601642-0fa6-4b68-b9cf-324c59e41637" volumeName="kubernetes.io/configmap/4d601642-0fa6-4b68-b9cf-324c59e41637-config-volume" seLinuxMountContext="" Nov 02 17:29:42.403516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403284 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61135006-920f-4fb5-8ba5-c98d6beae6cf" volumeName="kubernetes.io/projected/61135006-920f-4fb5-8ba5-c98d6beae6cf-kube-api-access-9fskz" seLinuxMountContext="" Nov 02 17:29:42.403516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403296 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="665ac753-596e-4d13-b8c1-7a1c7e791cf8" volumeName="kubernetes.io/projected/665ac753-596e-4d13-b8c1-7a1c7e791cf8-kube-api-access-24tpd" seLinuxMountContext="" Nov 02 17:29:42.403516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403340 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81019158-9e01-4dd1-a405-d1cf75d6dc78" volumeName="kubernetes.io/projected/81019158-9e01-4dd1-a405-d1cf75d6dc78-kube-api-access-pk945" seLinuxMountContext="" Nov 02 17:29:42.403516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403352 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9" volumeName="kubernetes.io/configmap/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-ovnkube-script-lib" seLinuxMountContext="" Nov 02 17:29:42.403516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403365 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9" volumeName="kubernetes.io/secret/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 17:29:42.403516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403375 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35ab9b40-d988-40ff-87c0-0cf822cfd863" volumeName="kubernetes.io/configmap/35ab9b40-d988-40ff-87c0-0cf822cfd863-cni-binary-copy" seLinuxMountContext="" Nov 02 17:29:42.403516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403386 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81019158-9e01-4dd1-a405-d1cf75d6dc78" volumeName="kubernetes.io/secret/81019158-9e01-4dd1-a405-d1cf75d6dc78-signing-key" seLinuxMountContext="" Nov 02 17:29:42.403516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403397 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d911b628-4ba2-4427-b229-7698d25c2e2c" volumeName="kubernetes.io/projected/d911b628-4ba2-4427-b229-7698d25c2e2c-kube-api-access-dmgzk" seLinuxMountContext="" Nov 02 17:29:42.403516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403410 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b" volumeName="kubernetes.io/projected/fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b-kube-api-access-8b57h" seLinuxMountContext="" Nov 02 17:29:42.403516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403421 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b5ee10f-87a1-4168-904e-ba233f304fe3" volumeName="kubernetes.io/secret/1b5ee10f-87a1-4168-904e-ba233f304fe3-agent-certs" seLinuxMountContext="" Nov 02 17:29:42.403516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403434 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d601642-0fa6-4b68-b9cf-324c59e41637" volumeName="kubernetes.io/projected/4d601642-0fa6-4b68-b9cf-324c59e41637-kube-api-access-wvnsx" seLinuxMountContext="" Nov 02 17:29:42.403516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403445 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61135006-920f-4fb5-8ba5-c98d6beae6cf" volumeName="kubernetes.io/empty-dir/61135006-920f-4fb5-8ba5-c98d6beae6cf-tmp" seLinuxMountContext="" Nov 02 17:29:42.403516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403457 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83c827e3-5338-49b5-97a1-56bdb5904500" volumeName="kubernetes.io/secret/83c827e3-5338-49b5-97a1-56bdb5904500-original-pull-secret" seLinuxMountContext="" Nov 02 17:29:42.403516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403467 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d911b628-4ba2-4427-b229-7698d25c2e2c" volumeName="kubernetes.io/secret/d911b628-4ba2-4427-b229-7698d25c2e2c-cert" seLinuxMountContext="" Nov 02 17:29:42.403516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403485 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="007d7841-411c-4b0f-b8aa-d36097b6bf31" volumeName="kubernetes.io/empty-dir/007d7841-411c-4b0f-b8aa-d36097b6bf31-data-volume" seLinuxMountContext="" Nov 02 17:29:42.403516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403514 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="007d7841-411c-4b0f-b8aa-d36097b6bf31" volumeName="kubernetes.io/projected/007d7841-411c-4b0f-b8aa-d36097b6bf31-kube-api-access-pmjd4" seLinuxMountContext="" Nov 02 17:29:42.403516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403526 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b0e99b5-4469-44ac-b241-a6edb43bc8ec" volumeName="kubernetes.io/projected/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-kube-api-access-zgkcf" seLinuxMountContext="" Nov 02 17:29:42.404280 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403539 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35ab9b40-d988-40ff-87c0-0cf822cfd863" volumeName="kubernetes.io/configmap/35ab9b40-d988-40ff-87c0-0cf822cfd863-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 17:29:42.404280 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403551 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d601642-0fa6-4b68-b9cf-324c59e41637" volumeName="kubernetes.io/empty-dir/4d601642-0fa6-4b68-b9cf-324c59e41637-tmp-dir" seLinuxMountContext="" Nov 02 17:29:42.404280 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403563 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c38cc772-7b2a-4a50-b230-e8b790647ae0" volumeName="kubernetes.io/configmap/c38cc772-7b2a-4a50-b230-e8b790647ae0-multus-daemon-config" seLinuxMountContext="" Nov 02 17:29:42.404280 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403575 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d09569ff-f9b3-42d4-b163-fae46612bffc" volumeName="kubernetes.io/projected/d09569ff-f9b3-42d4-b163-fae46612bffc-kube-api-access-cv9jt" seLinuxMountContext="" Nov 02 17:29:42.404280 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403596 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0aed3a0d-f6c0-4955-8d67-8b8b98bebe25" volumeName="kubernetes.io/projected/0aed3a0d-f6c0-4955-8d67-8b8b98bebe25-kube-api-access-cg9fq" seLinuxMountContext="" Nov 02 17:29:42.404280 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403605 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9" volumeName="kubernetes.io/configmap/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-ovnkube-config" seLinuxMountContext="" Nov 02 17:29:42.404280 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403612 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a4bf99d-1b04-48a4-bca9-dd3f484ad0a8" volumeName="kubernetes.io/projected/3a4bf99d-1b04-48a4-bca9-dd3f484ad0a8-kube-api-access-4qcnl" seLinuxMountContext="" Nov 02 17:29:42.404280 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403620 4471 reconstruct.go:97] "Volume reconstruction finished" Nov 02 17:29:42.404280 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.403627 4471 reconciler.go:29] "Reconciler: start to sync state" Nov 02 17:29:42.404717 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.404701 4471 factory.go:55] Registering systemd factory Nov 02 17:29:42.404760 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.404721 4471 factory.go:223] Registration of the systemd container factory successfully Nov 02 17:29:42.404904 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.404892 4471 factory.go:153] Registering CRI-O factory Nov 02 17:29:42.404966 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.404906 4471 factory.go:223] Registration of the crio container factory successfully Nov 02 17:29:42.404966 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.404959 4471 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 17:29:42.405118 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.404977 4471 factory.go:103] Registering Raw factory Nov 02 17:29:42.405118 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.404991 4471 manager.go:1199] Started watching for new ooms in manager Nov 02 17:29:42.405425 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.405410 4471 manager.go:319] Starting recovery of all containers Nov 02 17:29:42.431141 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.431093 4471 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 17:29:42.432590 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.432514 4471 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 17:29:42.432590 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.432549 4471 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 17:29:42.432590 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.432573 4471 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 17:29:42.432792 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:29:42.432613 4471 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 17:29:42.443630 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.443604 4471 generic.go:356] "Generic (PLEG): container finished" podID="041bb4fbe95805a40957f9ee210f4b23" containerID="801975d5c0d815c18e05c96a17e575f860556d85eea18898988f9a5c4e034aaf" exitCode=0 Nov 02 17:29:42.452347 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.452297 4471 generic.go:356] "Generic (PLEG): container finished" podID="35ab9b40-d988-40ff-87c0-0cf822cfd863" containerID="a784eba432f8f5c6606aebe621b584d33399b2e2dab6dc0754ea0ca1566d4b57" exitCode=0 Nov 02 17:29:42.452424 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.452348 4471 generic.go:356] "Generic (PLEG): container finished" podID="35ab9b40-d988-40ff-87c0-0cf822cfd863" containerID="031a156231396530cc0ef872f168e0f630f8fd31be92843f443460d9e32bc106" exitCode=0 Nov 02 17:29:42.452424 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.452359 4471 generic.go:356] "Generic (PLEG): container finished" podID="35ab9b40-d988-40ff-87c0-0cf822cfd863" containerID="0f36144c7e352ed92c970501fe7ef5d9618fde7de624c3c965a1a0ff7da335e4" exitCode=0 Nov 02 17:29:42.452424 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.452369 4471 generic.go:356] "Generic (PLEG): container finished" podID="35ab9b40-d988-40ff-87c0-0cf822cfd863" containerID="b01b88595570d99a443b64de4cc005a7d1aac11f485e46b813656c301f57bf7d" exitCode=0 Nov 02 17:29:42.452424 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.452379 4471 generic.go:356] "Generic (PLEG): container finished" podID="35ab9b40-d988-40ff-87c0-0cf822cfd863" containerID="5e07449ead7a871d88f150c122d63ee2b61fac4ef98ed9aa7e0e71b2846412ee" exitCode=0 Nov 02 17:29:42.452424 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.452387 4471 generic.go:356] "Generic (PLEG): container finished" podID="35ab9b40-d988-40ff-87c0-0cf822cfd863" containerID="668e946404e576494c689fbf72fbf622aade945eaf6fd64c6a40fe8b9cd20bdb" exitCode=0 Nov 02 17:29:42.454994 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.454960 4471 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 17:29:42.455084 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.455037 4471 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:29:42.467634 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.467613 4471 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:29:42.467872 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.467850 4471 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:29:42.468005 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.467972 4471 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:29:42.473287 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.473272 4471 manager.go:324] Recovery completed Nov 02 17:29:42.488623 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.488572 4471 cpu_manager.go:226] "Starting" policy="none" Nov 02 17:29:42.488623 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.488585 4471 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 17:29:42.488623 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.488601 4471 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 17:29:42.488770 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.488707 4471 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 17:29:42.488770 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.488718 4471 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 17:29:42.488770 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.488762 4471 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 17:29:42.488770 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.488767 4471 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 17:29:42.488883 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.488773 4471 policy_none.go:50] "Start" Nov 02 17:29:42.488883 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.488788 4471 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 17:29:42.488883 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.488797 4471 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 17:29:42.489345 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.489332 4471 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 17:29:42.489386 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.489345 4471 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 17:29:42.489386 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.489352 4471 policy_none.go:44] "Start" Nov 02 17:29:42.493367 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.493341 4471 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 17:29:42.495916 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.495903 4471 manager.go:341] "Starting Device Plugin manager" Nov 02 17:29:42.495965 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:29:42.495927 4471 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 17:29:42.495965 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.495938 4471 server.go:87] "Starting device plugin registration server" Nov 02 17:29:42.496260 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.496247 4471 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 17:29:42.496323 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.496263 4471 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 17:29:42.496381 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.496354 4471 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 17:29:42.496642 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.496466 4471 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 17:29:42.496642 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.496512 4471 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 17:29:42.496642 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.496519 4471 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 17:29:42.496642 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.496571 4471 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T17:29:42.496501346Z","UUID":"3ceed360-05e2-42e8-a95a-bbf60a2c2ac5","Handler":null,"Name":"","Endpoint":""} Nov 02 17:29:42.497903 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.497881 4471 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 17:29:42.498001 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:29:42.497904 4471 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 17:29:42.498001 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.497919 4471 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 17:29:42.533729 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.533691 4471 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-26.ec2.internal"] Nov 02 17:29:42.534123 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.534097 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal" event={"ID":"041bb4fbe95805a40957f9ee210f4b23","Type":"ContainerStarted","Data":"4023237f41cd5189da5a1e6681bb6cc6aadb6aa8418266ba87d27403f367bf10"} Nov 02 17:29:42.534161 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.534135 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal" event={"ID":"041bb4fbe95805a40957f9ee210f4b23","Type":"ContainerDied","Data":"801975d5c0d815c18e05c96a17e575f860556d85eea18898988f9a5c4e034aaf"} Nov 02 17:29:42.534161 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.534152 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal" event={"ID":"041bb4fbe95805a40957f9ee210f4b23","Type":"ContainerStarted","Data":"8d764223f8703f2c068280733e8df67e4b15cf89bec7e9f059b0baae9c2d2e91"} Nov 02 17:29:42.534233 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.534163 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-26.ec2.internal" event={"ID":"1b001a1b5ad5b960c3a58390390fe0a6","Type":"ContainerStarted","Data":"01c0c1a10661f609509a068380ec11c6e00c6deacbd11a9076c80fbd86d78691"} Nov 02 17:29:42.534233 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.534175 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-26.ec2.internal" event={"ID":"1b001a1b5ad5b960c3a58390390fe0a6","Type":"ContainerStarted","Data":"80dcc1078096451d0c8da3583979ee66c7a924f6d74409220f0c7ebee2ad3c46"} Nov 02 17:29:42.534294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.534278 4471 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal" Nov 02 17:29:42.534359 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.534348 4471 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-26.ec2.internal" Nov 02 17:29:42.558171 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.558155 4471 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:29:42.558217 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:29:42.558192 4471 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal" Nov 02 17:29:42.571373 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.571356 4471 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:29:42.571433 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:29:42.571393 4471 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-26.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-26.ec2.internal" Nov 02 17:29:42.596488 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.596471 4471 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:29:42.597770 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.597756 4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-26.ec2.internal" event="NodeHasSufficientMemory" Nov 02 17:29:42.597854 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.597784 4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-26.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 17:29:42.597854 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.597794 4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-26.ec2.internal" event="NodeHasSufficientPID" Nov 02 17:29:42.597854 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.597826 4471 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-26.ec2.internal" Nov 02 17:29:42.656398 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.656375 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/041bb4fbe95805a40957f9ee210f4b23-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal\" (UID: \"041bb4fbe95805a40957f9ee210f4b23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal" Nov 02 17:29:42.656488 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.656400 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/041bb4fbe95805a40957f9ee210f4b23-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal\" (UID: \"041bb4fbe95805a40957f9ee210f4b23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal" Nov 02 17:29:42.656488 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.656424 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1b001a1b5ad5b960c3a58390390fe0a6-config\") pod \"kube-apiserver-proxy-ip-10-0-142-26.ec2.internal\" (UID: \"1b001a1b5ad5b960c3a58390390fe0a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-26.ec2.internal" Nov 02 17:29:42.693516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.693497 4471 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-142-26.ec2.internal" Nov 02 17:29:42.693575 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.693553 4471 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-26.ec2.internal" Nov 02 17:29:42.757318 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.757240 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/041bb4fbe95805a40957f9ee210f4b23-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal\" (UID: \"041bb4fbe95805a40957f9ee210f4b23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal" Nov 02 17:29:42.757318 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.757270 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/041bb4fbe95805a40957f9ee210f4b23-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal\" (UID: \"041bb4fbe95805a40957f9ee210f4b23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal" Nov 02 17:29:42.757318 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.757290 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1b001a1b5ad5b960c3a58390390fe0a6-config\") pod \"kube-apiserver-proxy-ip-10-0-142-26.ec2.internal\" (UID: \"1b001a1b5ad5b960c3a58390390fe0a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-26.ec2.internal" Nov 02 17:29:42.757478 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.757363 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/041bb4fbe95805a40957f9ee210f4b23-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal\" (UID: \"041bb4fbe95805a40957f9ee210f4b23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal" Nov 02 17:29:42.757478 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.757416 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1b001a1b5ad5b960c3a58390390fe0a6-config\") pod \"kube-apiserver-proxy-ip-10-0-142-26.ec2.internal\" (UID: \"1b001a1b5ad5b960c3a58390390fe0a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-26.ec2.internal" Nov 02 17:29:42.757478 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:42.757448 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/041bb4fbe95805a40957f9ee210f4b23-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal\" (UID: \"041bb4fbe95805a40957f9ee210f4b23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal" Nov 02 17:29:43.382889 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.382856 4471 apiserver.go:52] "Watching apiserver" Nov 02 17:29:43.473572 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.473549 4471 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal" Nov 02 17:29:43.473709 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.473666 4471 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-26.ec2.internal" Nov 02 17:29:43.522461 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.522441 4471 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 17:29:43.522912 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.522893 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-26.ec2.internal","openshift-insights/insights-runtime-extractor-ccwcv","openshift-kube-storage-version-migrator/migrator-66df5d496b-bffdk","openshift-network-diagnostics/network-check-target-hsdff","kube-system/konnectivity-agent-rt5cp","openshift-cluster-node-tuning-operator/tuned-nsqnc","openshift-ingress-canary/ingress-canary-cqljg","openshift-multus/network-metrics-daemon-fdtgx","openshift-service-ca/service-ca-5fffc97bdf-mpdt5","openshift-multus/multus-additional-cni-plugins-phn5m","openshift-dns/node-resolver-22fc9","kube-system/global-pull-secret-syncer-l2q87","openshift-network-operator/iptables-alerter-bscjw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg","openshift-image-registry/node-ca-bz42x","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-8zq8j","openshift-multus/multus-7h4wg","openshift-dns/dns-default-bkdn9"] Nov 02 17:29:43.525720 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.525694 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:43.530042 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.530021 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bkdn9" Nov 02 17:29:43.530514 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.530497 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:43.530593 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.530575 4471 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:29:43.530670 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:29:43.530624 4471 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-26.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-26.ec2.internal" Nov 02 17:29:43.533869 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.533842 4471 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:29:43.533926 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:29:43.533897 4471 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-26.ec2.internal" Nov 02 17:29:43.566503 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.566481 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 17:29:43.566612 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.566597 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 17:29:43.566668 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.566636 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 17:29:43.566798 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.566779 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 17:29:43.566798 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.566789 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 17:29:43.566798 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.566798 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 17:29:43.566979 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.566781 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ppwdb\"" Nov 02 17:29:43.566979 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.566818 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 17:29:43.566979 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.566838 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zt2z5\"" Nov 02 17:29:43.566979 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.566851 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 17:29:43.571926 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.571910 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 17:29:43.575217 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.575200 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 17:29:43.575328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.575264 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 17:29:43.575328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.575291 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 17:29:43.575328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.575292 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 17:29:43.575808 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.575795 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 17:29:43.578522 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.578509 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 02 17:29:43.578558 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.578513 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 02 17:29:43.578808 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.578796 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 02 17:29:43.578846 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.578808 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 02 17:29:43.579118 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579096 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9lngf\"" Nov 02 17:29:43.579186 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579116 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4b9fl\"" Nov 02 17:29:43.579186 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579125 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 17:29:43.579186 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579129 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:29:43.579186 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579118 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 17:29:43.579186 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579145 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 17:29:43.579186 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579155 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 17:29:43.579419 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579190 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 17:29:43.579419 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579223 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 17:29:43.579419 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579268 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 17:29:43.579419 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579364 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qbqb7\"" Nov 02 17:29:43.579419 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579389 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 17:29:43.579564 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579428 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 17:29:43.579564 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579432 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 17:29:43.579564 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579449 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 17:29:43.579564 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579456 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 17:29:43.579564 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579451 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 17:29:43.579756 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579656 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 17:29:43.579756 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579739 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t2hrp\"" Nov 02 17:29:43.579854 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579759 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-w9x92\"" Nov 02 17:29:43.579854 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579762 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 17:29:43.579854 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579789 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 17:29:43.579854 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579807 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xdl26\"" Nov 02 17:29:43.579854 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579810 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-vqf9q\"" Nov 02 17:29:43.579854 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579815 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 17:29:43.579854 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579839 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 17:29:43.579854 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579845 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 17:29:43.580094 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579823 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 17:29:43.580094 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579807 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:29:43.580094 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579866 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fw7vt\"" Nov 02 17:29:43.580094 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579860 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 17:29:43.580094 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579864 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 17:29:43.580094 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579820 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rj9zp\"" Nov 02 17:29:43.580094 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579860 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 17:29:43.580094 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579907 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vjnnw\"" Nov 02 17:29:43.580094 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579876 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 17:29:43.580094 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579935 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5nstj\"" Nov 02 17:29:43.580094 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.579920 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jl8vw\"" Nov 02 17:29:43.584726 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.584712 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 17:29:43.584763 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.584743 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bl7zd\"" Nov 02 17:29:43.584791 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.584777 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-mwsx7\"" Nov 02 17:29:43.588537 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.588524 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nn2p2\"" Nov 02 17:29:43.592022 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.592010 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 17:29:43.603064 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.603048 4471 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 17:29:43.662574 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.662514 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-cnibin\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.662574 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.662544 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1b5ee10f-87a1-4168-904e-ba233f304fe3-agent-certs\") pod \"konnectivity-agent-rt5cp\" (UID: \"1b5ee10f-87a1-4168-904e-ba233f304fe3\") " pod="kube-system/konnectivity-agent-rt5cp" Nov 02 17:29:43.662574 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.662560 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-device-dir\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:43.662824 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.662576 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgkcf\" (UniqueName: \"kubernetes.io/projected/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-kube-api-access-zgkcf\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:43.662824 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.662594 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmjd4\" (UniqueName: \"kubernetes.io/projected/007d7841-411c-4b0f-b8aa-d36097b6bf31-kube-api-access-pmjd4\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:43.662824 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.662707 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-sysctl-d\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.662824 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.662726 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c38cc772-7b2a-4a50-b230-e8b790647ae0-cni-binary-copy\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.662824 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.662742 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35ab9b40-d988-40ff-87c0-0cf822cfd863-cni-binary-copy\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:43.662824 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.662757 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv9jt\" (UniqueName: \"kubernetes.io/projected/d09569ff-f9b3-42d4-b163-fae46612bffc-kube-api-access-cv9jt\") pod \"network-check-target-hsdff\" (UID: \"d09569ff-f9b3-42d4-b163-fae46612bffc\") " pod="openshift-network-diagnostics/network-check-target-hsdff" Nov 02 17:29:43.662824 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.662784 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-run-systemd\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.662824 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.662816 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-var-lib-openvswitch\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.663232 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.662840 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-ovn-node-metrics-cert\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.663232 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.662893 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4d601642-0fa6-4b68-b9cf-324c59e41637-config-volume\") pod \"dns-default-bkdn9\" (UID: \"4d601642-0fa6-4b68-b9cf-324c59e41637\") " pod="openshift-dns/dns-default-bkdn9" Nov 02 17:29:43.663232 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.662920 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-multus-cni-dir\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.663232 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.663071 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-log-socket\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.663232 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.663107 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-kubelet-dir\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:43.663232 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.663133 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-registration-dir\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:43.663232 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.663225 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/83c827e3-5338-49b5-97a1-56bdb5904500-kubelet-config\") pod \"global-pull-secret-syncer-l2q87\" (UID: \"83c827e3-5338-49b5-97a1-56bdb5904500\") " pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:43.663603 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.663270 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83c827e3-5338-49b5-97a1-56bdb5904500-original-pull-secret\") pod \"global-pull-secret-syncer-l2q87\" (UID: \"83c827e3-5338-49b5-97a1-56bdb5904500\") " pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:43.663603 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.663485 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1b5ee10f-87a1-4168-904e-ba233f304fe3-agent-certs\") pod \"konnectivity-agent-rt5cp\" (UID: \"1b5ee10f-87a1-4168-904e-ba233f304fe3\") " pod="kube-system/konnectivity-agent-rt5cp" Nov 02 17:29:43.663603 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.663498 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvnsx\" (UniqueName: \"kubernetes.io/projected/4d601642-0fa6-4b68-b9cf-324c59e41637-kube-api-access-wvnsx\") pod \"dns-default-bkdn9\" (UID: \"4d601642-0fa6-4b68-b9cf-324c59e41637\") " pod="openshift-dns/dns-default-bkdn9" Nov 02 17:29:43.663603 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.663541 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-modprobe-d\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.663603 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.663562 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-ovn-node-metrics-cert\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.663603 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.663581 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg9fq\" (UniqueName: \"kubernetes.io/projected/0aed3a0d-f6c0-4955-8d67-8b8b98bebe25-kube-api-access-cg9fq\") pod \"node-ca-bz42x\" (UID: \"0aed3a0d-f6c0-4955-8d67-8b8b98bebe25\") " pod="openshift-image-registry/node-ca-bz42x" Nov 02 17:29:43.663805 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.663606 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/665ac753-596e-4d13-b8c1-7a1c7e791cf8-hosts-file\") pod \"node-resolver-22fc9\" (UID: \"665ac753-596e-4d13-b8c1-7a1c7e791cf8\") " pod="openshift-dns/node-resolver-22fc9" Nov 02 17:29:43.663805 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.663639 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4d601642-0fa6-4b68-b9cf-324c59e41637-config-volume\") pod \"dns-default-bkdn9\" (UID: \"4d601642-0fa6-4b68-b9cf-324c59e41637\") " pod="openshift-dns/dns-default-bkdn9" Nov 02 17:29:43.663805 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.663659 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-var-lib-kubelet\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.663805 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.663676 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35ab9b40-d988-40ff-87c0-0cf822cfd863-cni-binary-copy\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:43.663805 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.663684 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83c827e3-5338-49b5-97a1-56bdb5904500-original-pull-secret\") pod \"global-pull-secret-syncer-l2q87\" (UID: \"83c827e3-5338-49b5-97a1-56bdb5904500\") " pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:43.663805 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.663676 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c38cc772-7b2a-4a50-b230-e8b790647ae0-cni-binary-copy\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.663805 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.663801 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-node-log\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.663989 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.663817 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d601642-0fa6-4b68-b9cf-324c59e41637-metrics-tls\") pod \"dns-default-bkdn9\" (UID: \"4d601642-0fa6-4b68-b9cf-324c59e41637\") " pod="openshift-dns/dns-default-bkdn9" Nov 02 17:29:43.663989 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.663833 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/007d7841-411c-4b0f-b8aa-d36097b6bf31-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:43.663989 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.663863 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-run-netns\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.663989 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.663892 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-sys\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.663989 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.663933 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b-iptables-alerter-script\") pod \"iptables-alerter-bscjw\" (UID: \"fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b\") " pod="openshift-network-operator/iptables-alerter-bscjw" Nov 02 17:29:43.663989 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.663955 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-etc-kubernetes\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.663989 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.663979 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-run-openvswitch\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.664210 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.664009 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/81019158-9e01-4dd1-a405-d1cf75d6dc78-signing-cabundle\") pod \"service-ca-5fffc97bdf-mpdt5\" (UID: \"81019158-9e01-4dd1-a405-d1cf75d6dc78\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mpdt5" Nov 02 17:29:43.664210 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.664038 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk945\" (UniqueName: \"kubernetes.io/projected/81019158-9e01-4dd1-a405-d1cf75d6dc78-kube-api-access-pk945\") pod \"service-ca-5fffc97bdf-mpdt5\" (UID: \"81019158-9e01-4dd1-a405-d1cf75d6dc78\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mpdt5" Nov 02 17:29:43.664210 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.664104 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-systemd\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.664210 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.664145 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-slash\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.664210 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.664186 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b-iptables-alerter-script\") pod \"iptables-alerter-bscjw\" (UID: \"fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b\") " pod="openshift-network-operator/iptables-alerter-bscjw" Nov 02 17:29:43.664210 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.664196 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-run\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.664417 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.664223 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61135006-920f-4fb5-8ba5-c98d6beae6cf-tmp\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.664417 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.664256 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/81019158-9e01-4dd1-a405-d1cf75d6dc78-signing-cabundle\") pod \"service-ca-5fffc97bdf-mpdt5\" (UID: \"81019158-9e01-4dd1-a405-d1cf75d6dc78\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mpdt5" Nov 02 17:29:43.664417 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.664261 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35ab9b40-d988-40ff-87c0-0cf822cfd863-tuning-conf-dir\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:43.664417 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.664296 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-sys-fs\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:43.664417 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.664322 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61135006-920f-4fb5-8ba5-c98d6beae6cf-tmp\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.664417 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.664353 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-host\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.664417 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.664373 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c38cc772-7b2a-4a50-b230-e8b790647ae0-multus-daemon-config\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.664417 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.664409 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-run-multus-certs\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.664710 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.664451 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35ab9b40-d988-40ff-87c0-0cf822cfd863-system-cni-dir\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:43.664710 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.664474 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35ab9b40-d988-40ff-87c0-0cf822cfd863-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:43.664710 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.664492 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rcc6\" (UniqueName: \"kubernetes.io/projected/35ab9b40-d988-40ff-87c0-0cf822cfd863-kube-api-access-4rcc6\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:43.664710 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.664508 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-etc-openvswitch\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.664710 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.664524 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c38cc772-7b2a-4a50-b230-e8b790647ae0-multus-daemon-config\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.664710 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.664531 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcgm2\" (UniqueName: \"kubernetes.io/projected/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-kube-api-access-jcgm2\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.664710 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.664567 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b-host-slash\") pod \"iptables-alerter-bscjw\" (UID: \"fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b\") " pod="openshift-network-operator/iptables-alerter-bscjw" Nov 02 17:29:43.664710 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.664681 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-var-lib-cni-multus\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.664710 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.664708 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-hostroot\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.665114 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.664728 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.665114 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.664788 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-env-overrides\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.665114 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.664811 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-socket-dir\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:43.665114 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.664834 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmgzk\" (UniqueName: \"kubernetes.io/projected/d911b628-4ba2-4427-b229-7698d25c2e2c-kube-api-access-dmgzk\") pod \"ingress-canary-cqljg\" (UID: \"d911b628-4ba2-4427-b229-7698d25c2e2c\") " pod="openshift-ingress-canary/ingress-canary-cqljg" Nov 02 17:29:43.665114 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.664838 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35ab9b40-d988-40ff-87c0-0cf822cfd863-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:43.665114 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.664849 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/81019158-9e01-4dd1-a405-d1cf75d6dc78-signing-key\") pod \"service-ca-5fffc97bdf-mpdt5\" (UID: \"81019158-9e01-4dd1-a405-d1cf75d6dc78\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mpdt5" Nov 02 17:29:43.665114 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.664864 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24tpd\" (UniqueName: \"kubernetes.io/projected/665ac753-596e-4d13-b8c1-7a1c7e791cf8-kube-api-access-24tpd\") pod \"node-resolver-22fc9\" (UID: \"665ac753-596e-4d13-b8c1-7a1c7e791cf8\") " pod="openshift-dns/node-resolver-22fc9" Nov 02 17:29:43.665114 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.664880 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-multus-socket-dir-parent\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.665114 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.664921 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-env-overrides\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.665114 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.664969 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-var-lib-cni-bin\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.665114 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665020 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-kubelet\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.665114 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665045 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-cni-netd\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.665114 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665087 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/007d7841-411c-4b0f-b8aa-d36097b6bf31-crio-socket\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:43.665114 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665107 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/81019158-9e01-4dd1-a405-d1cf75d6dc78-signing-key\") pod \"service-ca-5fffc97bdf-mpdt5\" (UID: \"81019158-9e01-4dd1-a405-d1cf75d6dc78\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mpdt5" Nov 02 17:29:43.665114 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665116 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-sysconfig\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.665675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665134 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fskz\" (UniqueName: \"kubernetes.io/projected/61135006-920f-4fb5-8ba5-c98d6beae6cf-kube-api-access-9fskz\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.665675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665150 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0aed3a0d-f6c0-4955-8d67-8b8b98bebe25-serviceca\") pod \"node-ca-bz42x\" (UID: \"0aed3a0d-f6c0-4955-8d67-8b8b98bebe25\") " pod="openshift-image-registry/node-ca-bz42x" Nov 02 17:29:43.665675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665165 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1b5ee10f-87a1-4168-904e-ba233f304fe3-konnectivity-ca\") pod \"konnectivity-agent-rt5cp\" (UID: \"1b5ee10f-87a1-4168-904e-ba233f304fe3\") " pod="kube-system/konnectivity-agent-rt5cp" Nov 02 17:29:43.665675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665183 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-ovnkube-config\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.665675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665197 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d911b628-4ba2-4427-b229-7698d25c2e2c-cert\") pod \"ingress-canary-cqljg\" (UID: \"d911b628-4ba2-4427-b229-7698d25c2e2c\") " pod="openshift-ingress-canary/ingress-canary-cqljg" Nov 02 17:29:43.665675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665217 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d601642-0fa6-4b68-b9cf-324c59e41637-tmp-dir\") pod \"dns-default-bkdn9\" (UID: \"4d601642-0fa6-4b68-b9cf-324c59e41637\") " pod="openshift-dns/dns-default-bkdn9" Nov 02 17:29:43.665675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665244 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-var-lib-kubelet\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.665675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665296 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-run-k8s-cni-cncf-io\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.665675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665355 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cb013c0-4887-488b-867f-841fb6b01ca5-metrics-certs\") pod \"network-metrics-daemon-fdtgx\" (UID: \"5cb013c0-4887-488b-867f-841fb6b01ca5\") " pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:43.665675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665362 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1b5ee10f-87a1-4168-904e-ba233f304fe3-konnectivity-ca\") pod \"konnectivity-agent-rt5cp\" (UID: \"1b5ee10f-87a1-4168-904e-ba233f304fe3\") " pod="kube-system/konnectivity-agent-rt5cp" Nov 02 17:29:43.665675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665359 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d601642-0fa6-4b68-b9cf-324c59e41637-tmp-dir\") pod \"dns-default-bkdn9\" (UID: \"4d601642-0fa6-4b68-b9cf-324c59e41637\") " pod="openshift-dns/dns-default-bkdn9" Nov 02 17:29:43.665675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665382 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-systemd-units\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.665675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665414 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/83c827e3-5338-49b5-97a1-56bdb5904500-dbus\") pod \"global-pull-secret-syncer-l2q87\" (UID: \"83c827e3-5338-49b5-97a1-56bdb5904500\") " pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:43.665675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665438 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-kubernetes\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.665675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665440 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d911b628-4ba2-4427-b229-7698d25c2e2c-cert\") pod \"ingress-canary-cqljg\" (UID: \"d911b628-4ba2-4427-b229-7698d25c2e2c\") " pod="openshift-ingress-canary/ingress-canary-cqljg" Nov 02 17:29:43.665675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665438 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-ovnkube-config\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.665675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665462 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-lib-modules\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.665675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665492 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0aed3a0d-f6c0-4955-8d67-8b8b98bebe25-host\") pod \"node-ca-bz42x\" (UID: \"0aed3a0d-f6c0-4955-8d67-8b8b98bebe25\") " pod="openshift-image-registry/node-ca-bz42x" Nov 02 17:29:43.665675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665546 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qcnl\" (UniqueName: \"kubernetes.io/projected/3a4bf99d-1b04-48a4-bca9-dd3f484ad0a8-kube-api-access-4qcnl\") pod \"migrator-66df5d496b-bffdk\" (UID: \"3a4bf99d-1b04-48a4-bca9-dd3f484ad0a8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bffdk" Nov 02 17:29:43.665675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665564 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-multus-conf-dir\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.665675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665596 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-run-ovn-kubernetes\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.665675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665612 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-cni-bin\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.665675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665632 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-etc-selinux\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:43.665675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665663 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/007d7841-411c-4b0f-b8aa-d36097b6bf31-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:43.665675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665689 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-sysctl-conf\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.666490 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665719 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-tuned\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.666490 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665752 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/665ac753-596e-4d13-b8c1-7a1c7e791cf8-tmp-dir\") pod \"node-resolver-22fc9\" (UID: \"665ac753-596e-4d13-b8c1-7a1c7e791cf8\") " pod="openshift-dns/node-resolver-22fc9" Nov 02 17:29:43.666490 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665774 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-tuned\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.666490 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665789 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0aed3a0d-f6c0-4955-8d67-8b8b98bebe25-serviceca\") pod \"node-ca-bz42x\" (UID: \"0aed3a0d-f6c0-4955-8d67-8b8b98bebe25\") " pod="openshift-image-registry/node-ca-bz42x" Nov 02 17:29:43.666490 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665821 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-system-cni-dir\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.666490 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665855 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/665ac753-596e-4d13-b8c1-7a1c7e791cf8-tmp-dir\") pod \"node-resolver-22fc9\" (UID: \"665ac753-596e-4d13-b8c1-7a1c7e791cf8\") " pod="openshift-dns/node-resolver-22fc9" Nov 02 17:29:43.666490 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665879 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-run-netns\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.666490 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665903 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35ab9b40-d988-40ff-87c0-0cf822cfd863-cnibin\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:43.666490 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665931 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35ab9b40-d988-40ff-87c0-0cf822cfd863-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:43.666490 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665947 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/007d7841-411c-4b0f-b8aa-d36097b6bf31-data-volume\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:43.666490 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.665992 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88l9x\" (UniqueName: \"kubernetes.io/projected/c38cc772-7b2a-4a50-b230-e8b790647ae0-kube-api-access-88l9x\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.666490 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.666036 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/007d7841-411c-4b0f-b8aa-d36097b6bf31-data-volume\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:43.666490 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.666036 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgsrn\" (UniqueName: \"kubernetes.io/projected/5cb013c0-4887-488b-867f-841fb6b01ca5-kube-api-access-dgsrn\") pod \"network-metrics-daemon-fdtgx\" (UID: \"5cb013c0-4887-488b-867f-841fb6b01ca5\") " pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:43.666490 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.666076 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-run-ovn\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.666490 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.666106 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-ovnkube-script-lib\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.666490 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.666131 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b57h\" (UniqueName: \"kubernetes.io/projected/fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b-kube-api-access-8b57h\") pod \"iptables-alerter-bscjw\" (UID: \"fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b\") " pod="openshift-network-operator/iptables-alerter-bscjw" Nov 02 17:29:43.666490 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.666161 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35ab9b40-d988-40ff-87c0-0cf822cfd863-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:43.666490 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.666331 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-ovnkube-script-lib\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.688482 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.688458 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvnsx\" (UniqueName: \"kubernetes.io/projected/4d601642-0fa6-4b68-b9cf-324c59e41637-kube-api-access-wvnsx\") pod \"dns-default-bkdn9\" (UID: \"4d601642-0fa6-4b68-b9cf-324c59e41637\") " pod="openshift-dns/dns-default-bkdn9" Nov 02 17:29:43.688891 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.688872 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv9jt\" (UniqueName: \"kubernetes.io/projected/d09569ff-f9b3-42d4-b163-fae46612bffc-kube-api-access-cv9jt\") pod \"network-check-target-hsdff\" (UID: \"d09569ff-f9b3-42d4-b163-fae46612bffc\") " pod="openshift-network-diagnostics/network-check-target-hsdff" Nov 02 17:29:43.699341 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.699279 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmgzk\" (UniqueName: \"kubernetes.io/projected/d911b628-4ba2-4427-b229-7698d25c2e2c-kube-api-access-dmgzk\") pod \"ingress-canary-cqljg\" (UID: \"d911b628-4ba2-4427-b229-7698d25c2e2c\") " pod="openshift-ingress-canary/ingress-canary-cqljg" Nov 02 17:29:43.699445 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.699299 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcgm2\" (UniqueName: \"kubernetes.io/projected/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-kube-api-access-jcgm2\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.699445 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.699385 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk945\" (UniqueName: \"kubernetes.io/projected/81019158-9e01-4dd1-a405-d1cf75d6dc78-kube-api-access-pk945\") pod \"service-ca-5fffc97bdf-mpdt5\" (UID: \"81019158-9e01-4dd1-a405-d1cf75d6dc78\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mpdt5" Nov 02 17:29:43.701413 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.701393 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rcc6\" (UniqueName: \"kubernetes.io/projected/35ab9b40-d988-40ff-87c0-0cf822cfd863-kube-api-access-4rcc6\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:43.704777 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.704756 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg9fq\" (UniqueName: \"kubernetes.io/projected/0aed3a0d-f6c0-4955-8d67-8b8b98bebe25-kube-api-access-cg9fq\") pod \"node-ca-bz42x\" (UID: \"0aed3a0d-f6c0-4955-8d67-8b8b98bebe25\") " pod="openshift-image-registry/node-ca-bz42x" Nov 02 17:29:43.705605 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.705588 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24tpd\" (UniqueName: \"kubernetes.io/projected/665ac753-596e-4d13-b8c1-7a1c7e791cf8-kube-api-access-24tpd\") pod \"node-resolver-22fc9\" (UID: \"665ac753-596e-4d13-b8c1-7a1c7e791cf8\") " pod="openshift-dns/node-resolver-22fc9" Nov 02 17:29:43.709916 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.709896 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b57h\" (UniqueName: \"kubernetes.io/projected/fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b-kube-api-access-8b57h\") pod \"iptables-alerter-bscjw\" (UID: \"fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b\") " pod="openshift-network-operator/iptables-alerter-bscjw" Nov 02 17:29:43.710067 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.710050 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmjd4\" (UniqueName: \"kubernetes.io/projected/007d7841-411c-4b0f-b8aa-d36097b6bf31-kube-api-access-pmjd4\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:43.710825 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.710807 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgsrn\" (UniqueName: \"kubernetes.io/projected/5cb013c0-4887-488b-867f-841fb6b01ca5-kube-api-access-dgsrn\") pod \"network-metrics-daemon-fdtgx\" (UID: \"5cb013c0-4887-488b-867f-841fb6b01ca5\") " pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:43.710987 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.710971 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qcnl\" (UniqueName: \"kubernetes.io/projected/3a4bf99d-1b04-48a4-bca9-dd3f484ad0a8-kube-api-access-4qcnl\") pod \"migrator-66df5d496b-bffdk\" (UID: \"3a4bf99d-1b04-48a4-bca9-dd3f484ad0a8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bffdk" Nov 02 17:29:43.713032 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.713016 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fskz\" (UniqueName: \"kubernetes.io/projected/61135006-920f-4fb5-8ba5-c98d6beae6cf-kube-api-access-9fskz\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.715471 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.715452 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88l9x\" (UniqueName: \"kubernetes.io/projected/c38cc772-7b2a-4a50-b230-e8b790647ae0-kube-api-access-88l9x\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.717996 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.717979 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgkcf\" (UniqueName: \"kubernetes.io/projected/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-kube-api-access-zgkcf\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:43.766554 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.766530 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-host\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.766554 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.766556 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-run-multus-certs\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.766711 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.766572 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35ab9b40-d988-40ff-87c0-0cf822cfd863-system-cni-dir\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:43.766711 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.766587 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-etc-openvswitch\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.766711 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.766622 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b-host-slash\") pod \"iptables-alerter-bscjw\" (UID: \"fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b\") " pod="openshift-network-operator/iptables-alerter-bscjw" Nov 02 17:29:43.766711 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.766638 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-etc-openvswitch\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.766711 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.766650 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-var-lib-cni-multus\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.766711 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.766657 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35ab9b40-d988-40ff-87c0-0cf822cfd863-system-cni-dir\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:43.766711 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.766680 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-run-multus-certs\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.766711 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.766667 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-hostroot\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.766711 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.766697 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-var-lib-cni-multus\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.766711 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.766659 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-host\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.766711 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.766709 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.767218 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.766726 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-socket-dir\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:43.767218 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.766727 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b-host-slash\") pod \"iptables-alerter-bscjw\" (UID: \"fe1a6d6b-5d70-4e68-ad95-1a7b7943b76b\") " pod="openshift-network-operator/iptables-alerter-bscjw" Nov 02 17:29:43.767218 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.766746 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-hostroot\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.767218 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.766779 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-socket-dir\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:43.767218 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.766787 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.767218 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.766834 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-multus-socket-dir-parent\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.767218 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.766923 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-multus-socket-dir-parent\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.767218 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.767187 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-var-lib-cni-bin\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.767580 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.767242 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-kubelet\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.767580 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.767245 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-var-lib-cni-bin\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.767580 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.767299 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-cni-netd\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.767580 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.767355 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/007d7841-411c-4b0f-b8aa-d36097b6bf31-crio-socket\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:43.767580 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.767368 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-kubelet\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.767580 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.767408 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-cni-netd\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.767580 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.767458 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-sysconfig\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.767580 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.767521 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-sysconfig\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.767580 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.767524 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/007d7841-411c-4b0f-b8aa-d36097b6bf31-crio-socket\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:43.767580 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.767573 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-var-lib-kubelet\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.767997 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.767599 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-run-k8s-cni-cncf-io\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.767997 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.767643 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cb013c0-4887-488b-867f-841fb6b01ca5-metrics-certs\") pod \"network-metrics-daemon-fdtgx\" (UID: \"5cb013c0-4887-488b-867f-841fb6b01ca5\") " pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:43.767997 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.767677 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-systemd-units\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.767997 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.767706 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-run-k8s-cni-cncf-io\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.767997 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.767646 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-var-lib-kubelet\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.767997 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.767720 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/83c827e3-5338-49b5-97a1-56bdb5904500-dbus\") pod \"global-pull-secret-syncer-l2q87\" (UID: \"83c827e3-5338-49b5-97a1-56bdb5904500\") " pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:43.767997 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.767753 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-kubernetes\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.767997 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.767791 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-lib-modules\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.767997 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.767818 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0aed3a0d-f6c0-4955-8d67-8b8b98bebe25-host\") pod \"node-ca-bz42x\" (UID: \"0aed3a0d-f6c0-4955-8d67-8b8b98bebe25\") " pod="openshift-image-registry/node-ca-bz42x" Nov 02 17:29:43.767997 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:29:43.767830 4471 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 17:29:43.767997 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.767862 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-multus-conf-dir\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.767997 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.767893 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-run-ovn-kubernetes\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.767997 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:29:43.767920 4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cb013c0-4887-488b-867f-841fb6b01ca5-metrics-certs podName:5cb013c0-4887-488b-867f-841fb6b01ca5 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:44.267893549 +0000 UTC m=+1.963249021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5cb013c0-4887-488b-867f-841fb6b01ca5-metrics-certs") pod "network-metrics-daemon-fdtgx" (UID: "5cb013c0-4887-488b-867f-841fb6b01ca5") : secret "metrics-daemon-secret" not found Nov 02 17:29:43.767997 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.767951 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-cni-bin\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.767997 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.767956 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-run-ovn-kubernetes\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.767997 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.767986 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-etc-selinux\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:43.768659 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.768026 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-systemd-units\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.768659 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.768040 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/007d7841-411c-4b0f-b8aa-d36097b6bf31-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:43.768659 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.768094 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-sysctl-conf\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.768659 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.768112 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/83c827e3-5338-49b5-97a1-56bdb5904500-dbus\") pod \"global-pull-secret-syncer-l2q87\" (UID: \"83c827e3-5338-49b5-97a1-56bdb5904500\") " pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:43.768659 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.768190 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-kubernetes\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.768659 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.768343 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-lib-modules\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.768659 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.768458 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0aed3a0d-f6c0-4955-8d67-8b8b98bebe25-host\") pod \"node-ca-bz42x\" (UID: \"0aed3a0d-f6c0-4955-8d67-8b8b98bebe25\") " pod="openshift-image-registry/node-ca-bz42x" Nov 02 17:29:43.768659 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.768507 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-multus-conf-dir\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.768659 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.768583 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-etc-selinux\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:43.769013 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.768842 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-cni-bin\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.769372 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.769354 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-system-cni-dir\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.769636 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.769450 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/007d7841-411c-4b0f-b8aa-d36097b6bf31-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:43.769723 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.768124 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-system-cni-dir\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.769723 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.769673 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-run-netns\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.769723 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.769708 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35ab9b40-d988-40ff-87c0-0cf822cfd863-cnibin\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:43.769851 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.769741 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-run-ovn\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.769851 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.769773 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-cnibin\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.770263 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.770243 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-device-dir\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:43.770346 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.770285 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-sysctl-d\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.770346 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.770333 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-run-systemd\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.770443 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.770362 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-var-lib-openvswitch\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.770443 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.770391 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-multus-cni-dir\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.770443 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.770412 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-log-socket\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.770571 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.770440 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-kubelet-dir\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:43.770571 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.770469 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-registration-dir\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:43.770571 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.770496 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/83c827e3-5338-49b5-97a1-56bdb5904500-kubelet-config\") pod \"global-pull-secret-syncer-l2q87\" (UID: \"83c827e3-5338-49b5-97a1-56bdb5904500\") " pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:43.770571 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.770521 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-modprobe-d\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.770571 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.770550 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/665ac753-596e-4d13-b8c1-7a1c7e791cf8-hosts-file\") pod \"node-resolver-22fc9\" (UID: \"665ac753-596e-4d13-b8c1-7a1c7e791cf8\") " pod="openshift-dns/node-resolver-22fc9" Nov 02 17:29:43.770781 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.770579 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-var-lib-kubelet\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.770781 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.770606 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-node-log\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.770781 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.770633 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d601642-0fa6-4b68-b9cf-324c59e41637-metrics-tls\") pod \"dns-default-bkdn9\" (UID: \"4d601642-0fa6-4b68-b9cf-324c59e41637\") " pod="openshift-dns/dns-default-bkdn9" Nov 02 17:29:43.770781 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.770658 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/007d7841-411c-4b0f-b8aa-d36097b6bf31-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:43.770781 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.770687 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-run-netns\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.770781 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.770740 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-run-netns\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.770781 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.770754 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-log-socket\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.771091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.770778 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-sysctl-conf\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.771091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.770792 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-device-dir\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:43.771091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.770806 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/665ac753-596e-4d13-b8c1-7a1c7e791cf8-hosts-file\") pod \"node-resolver-22fc9\" (UID: \"665ac753-596e-4d13-b8c1-7a1c7e791cf8\") " pod="openshift-dns/node-resolver-22fc9" Nov 02 17:29:43.771091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.770823 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-multus-cni-dir\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.771091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.770840 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-registration-dir\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:43.771091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.770879 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-var-lib-openvswitch\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.771091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.770906 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-run-systemd\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.771091 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:29:43.770918 4471 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 17:29:43.771091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.770922 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-run-netns\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.771091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.770913 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-sys\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.771091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.770932 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-modprobe-d\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.771091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.770977 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-node-log\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.771091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.770968 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-etc-kubernetes\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.771091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.770981 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-sys\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.771091 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:29:43.770987 4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d601642-0fa6-4b68-b9cf-324c59e41637-metrics-tls podName:4d601642-0fa6-4b68-b9cf-324c59e41637 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:44.27096956 +0000 UTC m=+1.966325019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4d601642-0fa6-4b68-b9cf-324c59e41637-metrics-tls") pod "dns-default-bkdn9" (UID: "4d601642-0fa6-4b68-b9cf-324c59e41637") : secret "dns-default-metrics-tls" not found Nov 02 17:29:43.771091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.771020 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-run-ovn\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.771091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.771068 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-cnibin\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.771091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.771073 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-host-var-lib-kubelet\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.771091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.771091 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c38cc772-7b2a-4a50-b230-e8b790647ae0-etc-kubernetes\") pod \"multus-7h4wg\" (UID: \"c38cc772-7b2a-4a50-b230-e8b790647ae0\") " pod="openshift-multus/multus-7h4wg" Nov 02 17:29:43.771784 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.771124 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/83c827e3-5338-49b5-97a1-56bdb5904500-kubelet-config\") pod \"global-pull-secret-syncer-l2q87\" (UID: \"83c827e3-5338-49b5-97a1-56bdb5904500\") " pod="kube-system/global-pull-secret-syncer-l2q87" Nov 02 17:29:43.771784 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.771153 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-kubelet-dir\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:43.771784 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.771156 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-run-openvswitch\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.771784 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.771187 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-sysctl-d\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.771784 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.771217 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35ab9b40-d988-40ff-87c0-0cf822cfd863-cnibin\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:43.771784 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.771223 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-run-openvswitch\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.771784 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.771229 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-systemd\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.771784 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:29:43.771224 4471 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 17:29:43.771784 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.771333 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-slash\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.771784 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.771347 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-etc-systemd\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.771784 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:29:43.771350 4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/007d7841-411c-4b0f-b8aa-d36097b6bf31-insights-runtime-extractor-tls podName:007d7841-411c-4b0f-b8aa-d36097b6bf31 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:44.271335438 +0000 UTC m=+1.966690904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/007d7841-411c-4b0f-b8aa-d36097b6bf31-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ccwcv" (UID: "007d7841-411c-4b0f-b8aa-d36097b6bf31") : secret "insights-runtime-extractor-tls" not found Nov 02 17:29:43.771784 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.771378 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9-host-slash\") pod \"ovnkube-node-8zq8j\" (UID: \"15e5d6c0-3d7a-4c21-b44a-f0fd09f3b2a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:43.771784 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.771388 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-run\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.771784 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.771747 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35ab9b40-d988-40ff-87c0-0cf822cfd863-tuning-conf-dir\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:43.771784 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.771789 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-sys-fs\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:43.771784 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.771790 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/61135006-920f-4fb5-8ba5-c98d6beae6cf-run\") pod \"tuned-nsqnc\" (UID: \"61135006-920f-4fb5-8ba5-c98d6beae6cf\") " pod="openshift-cluster-node-tuning-operator/tuned-nsqnc" Nov 02 17:29:43.772217 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.771790 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35ab9b40-d988-40ff-87c0-0cf822cfd863-tuning-conf-dir\") pod \"multus-additional-cni-plugins-phn5m\" (UID: \"35ab9b40-d988-40ff-87c0-0cf822cfd863\") " pod="openshift-multus/multus-additional-cni-plugins-phn5m" Nov 02 17:29:43.772217 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.771854 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0b0e99b5-4469-44ac-b241-a6edb43bc8ec-sys-fs\") pod \"aws-ebs-csi-driver-node-59dmg\" (UID: \"0b0e99b5-4469-44ac-b241-a6edb43bc8ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59dmg" Nov 02 17:29:43.825467 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.825427 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rt5cp_1b5ee10f-87a1-4168-904e-ba233f304fe3/konnectivity-agent/0.log" limit=1024 Nov 02 17:29:43.832561 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.832544 4471 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:29:43.994134 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:43.994069 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:44.155353 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:44.155328 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-22fc9_665ac753-596e-4d13-b8c1-7a1c7e791cf8/dns-node-resolver/0.log" Nov 02 17:29:44.274831 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:44.274807 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d601642-0fa6-4b68-b9cf-324c59e41637-metrics-tls\") pod \"dns-default-bkdn9\" (UID: \"4d601642-0fa6-4b68-b9cf-324c59e41637\") " pod="openshift-dns/dns-default-bkdn9" Nov 02 17:29:44.274995 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:44.274839 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/007d7841-411c-4b0f-b8aa-d36097b6bf31-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:44.274995 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:44.274873 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cb013c0-4887-488b-867f-841fb6b01ca5-metrics-certs\") pod \"network-metrics-daemon-fdtgx\" (UID: \"5cb013c0-4887-488b-867f-841fb6b01ca5\") " pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:44.274995 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:29:44.274955 4471 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 17:29:44.274995 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:29:44.274957 4471 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 17:29:44.274995 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:29:44.274998 4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cb013c0-4887-488b-867f-841fb6b01ca5-metrics-certs podName:5cb013c0-4887-488b-867f-841fb6b01ca5 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:45.274986068 +0000 UTC m=+2.970341519 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5cb013c0-4887-488b-867f-841fb6b01ca5-metrics-certs") pod "network-metrics-daemon-fdtgx" (UID: "5cb013c0-4887-488b-867f-841fb6b01ca5") : secret "metrics-daemon-secret" not found Nov 02 17:29:44.275178 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:29:44.275016 4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d601642-0fa6-4b68-b9cf-324c59e41637-metrics-tls podName:4d601642-0fa6-4b68-b9cf-324c59e41637 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:45.275002802 +0000 UTC m=+2.970358254 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4d601642-0fa6-4b68-b9cf-324c59e41637-metrics-tls") pod "dns-default-bkdn9" (UID: "4d601642-0fa6-4b68-b9cf-324c59e41637") : secret "dns-default-metrics-tls" not found Nov 02 17:29:44.275178 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:29:44.274964 4471 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 17:29:44.275178 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:29:44.275047 4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/007d7841-411c-4b0f-b8aa-d36097b6bf31-insights-runtime-extractor-tls podName:007d7841-411c-4b0f-b8aa-d36097b6bf31 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:45.275040202 +0000 UTC m=+2.970395653 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/007d7841-411c-4b0f-b8aa-d36097b6bf31-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ccwcv" (UID: "007d7841-411c-4b0f-b8aa-d36097b6bf31") : secret "insights-runtime-extractor-tls" not found Nov 02 17:29:44.477198 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:44.477161 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-mpdt5" event={"ID":"81019158-9e01-4dd1-a405-d1cf75d6dc78","Type":"ContainerStarted","Data":"1ae69ea02c0d8230659b117cbf35886051b95d175a70af0234001a9cb6d5462d"} Nov 02 17:29:45.155329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:45.154372 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bz42x_0aed3a0d-f6c0-4955-8d67-8b8b98bebe25/node-ca/0.log" Nov 02 17:29:45.281972 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:45.281939 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d601642-0fa6-4b68-b9cf-324c59e41637-metrics-tls\") pod \"dns-default-bkdn9\" (UID: \"4d601642-0fa6-4b68-b9cf-324c59e41637\") " pod="openshift-dns/dns-default-bkdn9" Nov 02 17:29:45.281972 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:45.281978 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/007d7841-411c-4b0f-b8aa-d36097b6bf31-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:45.282213 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:45.282013 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cb013c0-4887-488b-867f-841fb6b01ca5-metrics-certs\") pod \"network-metrics-daemon-fdtgx\" (UID: \"5cb013c0-4887-488b-867f-841fb6b01ca5\") " pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:45.282213 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:29:45.282111 4471 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 17:29:45.282213 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:29:45.282168 4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cb013c0-4887-488b-867f-841fb6b01ca5-metrics-certs podName:5cb013c0-4887-488b-867f-841fb6b01ca5 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:47.282155019 +0000 UTC m=+4.977510470 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5cb013c0-4887-488b-867f-841fb6b01ca5-metrics-certs") pod "network-metrics-daemon-fdtgx" (UID: "5cb013c0-4887-488b-867f-841fb6b01ca5") : secret "metrics-daemon-secret" not found Nov 02 17:29:45.282530 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:29:45.282499 4471 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 17:29:45.282530 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:29:45.282524 4471 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 17:29:45.282692 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:29:45.282586 4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/007d7841-411c-4b0f-b8aa-d36097b6bf31-insights-runtime-extractor-tls podName:007d7841-411c-4b0f-b8aa-d36097b6bf31 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:47.282568384 +0000 UTC m=+4.977923840 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/007d7841-411c-4b0f-b8aa-d36097b6bf31-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ccwcv" (UID: "007d7841-411c-4b0f-b8aa-d36097b6bf31") : secret "insights-runtime-extractor-tls" not found Nov 02 17:29:45.282692 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:29:45.282606 4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d601642-0fa6-4b68-b9cf-324c59e41637-metrics-tls podName:4d601642-0fa6-4b68-b9cf-324c59e41637 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:47.282597072 +0000 UTC m=+4.977952531 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4d601642-0fa6-4b68-b9cf-324c59e41637-metrics-tls") pod "dns-default-bkdn9" (UID: "4d601642-0fa6-4b68-b9cf-324c59e41637") : secret "dns-default-metrics-tls" not found Nov 02 17:29:46.148957 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:46.148925 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cqljg_d911b628-4ba2-4427-b229-7698d25c2e2c/serve-healthcheck-canary/0.log" Nov 02 17:29:46.550393 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:46.550362 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-bffdk_3a4bf99d-1b04-48a4-bca9-dd3f484ad0a8/migrator/0.log" Nov 02 17:29:47.293691 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:47.293645 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d601642-0fa6-4b68-b9cf-324c59e41637-metrics-tls\") pod \"dns-default-bkdn9\" (UID: \"4d601642-0fa6-4b68-b9cf-324c59e41637\") " pod="openshift-dns/dns-default-bkdn9" Nov 02 17:29:47.293691 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:47.293697 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/007d7841-411c-4b0f-b8aa-d36097b6bf31-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:47.294149 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:29:47.293796 4471 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 17:29:47.294149 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:47.293827 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cb013c0-4887-488b-867f-841fb6b01ca5-metrics-certs\") pod \"network-metrics-daemon-fdtgx\" (UID: \"5cb013c0-4887-488b-867f-841fb6b01ca5\") " pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:47.294149 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:29:47.293844 4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/007d7841-411c-4b0f-b8aa-d36097b6bf31-insights-runtime-extractor-tls podName:007d7841-411c-4b0f-b8aa-d36097b6bf31 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:51.293833045 +0000 UTC m=+8.989188496 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/007d7841-411c-4b0f-b8aa-d36097b6bf31-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ccwcv" (UID: "007d7841-411c-4b0f-b8aa-d36097b6bf31") : secret "insights-runtime-extractor-tls" not found Nov 02 17:29:47.294149 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:47.294057 4471 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 17:29:47.296125 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:47.296096 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d601642-0fa6-4b68-b9cf-324c59e41637-metrics-tls\") pod \"dns-default-bkdn9\" (UID: \"4d601642-0fa6-4b68-b9cf-324c59e41637\") " pod="openshift-dns/dns-default-bkdn9" Nov 02 17:29:47.296125 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:47.296120 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cb013c0-4887-488b-867f-841fb6b01ca5-metrics-certs\") pod \"network-metrics-daemon-fdtgx\" (UID: \"5cb013c0-4887-488b-867f-841fb6b01ca5\") " pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:47.426620 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:47.426589 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdtgx" Nov 02 17:29:47.430975 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:47.430953 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bkdn9" Nov 02 17:29:47.754896 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:47.754870 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bkdn9"] Nov 02 17:29:47.756016 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:47.755990 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d601642_0fa6_4b68_b9cf_324c59e41637.slice/crio-245354c277ba6c119af6b3d1145d19bc407d8fd9b54850f2cdf5b097e0cf412d WatchSource:0}: Error finding container 245354c277ba6c119af6b3d1145d19bc407d8fd9b54850f2cdf5b097e0cf412d: Status 404 returned error can't find the container with id 245354c277ba6c119af6b3d1145d19bc407d8fd9b54850f2cdf5b097e0cf412d Nov 02 17:29:47.772714 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:47.772692 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fdtgx"] Nov 02 17:29:47.773286 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:47.773265 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cb013c0_4887_488b_867f_841fb6b01ca5.slice/crio-e745b82c4ed139cbdbb253ee9a8c2d1b5c9faaf43bcfe5d2323b1562520518b7 WatchSource:0}: Error finding container e745b82c4ed139cbdbb253ee9a8c2d1b5c9faaf43bcfe5d2323b1562520518b7: Status 404 returned error can't find the container with id e745b82c4ed139cbdbb253ee9a8c2d1b5c9faaf43bcfe5d2323b1562520518b7 Nov 02 17:29:48.489760 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:48.489722 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bkdn9" event={"ID":"4d601642-0fa6-4b68-b9cf-324c59e41637","Type":"ContainerStarted","Data":"245354c277ba6c119af6b3d1145d19bc407d8fd9b54850f2cdf5b097e0cf412d"} Nov 02 17:29:48.491138 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:48.491113 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fdtgx" event={"ID":"5cb013c0-4887-488b-867f-841fb6b01ca5","Type":"ContainerStarted","Data":"e745b82c4ed139cbdbb253ee9a8c2d1b5c9faaf43bcfe5d2323b1562520518b7"} Nov 02 17:29:49.498748 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:49.498667 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bkdn9" event={"ID":"4d601642-0fa6-4b68-b9cf-324c59e41637","Type":"ContainerStarted","Data":"01873ed96c6103bfe6e9aa1cf383f9444b33d951695d9419f943125963c89299"} Nov 02 17:29:49.499137 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:49.498765 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bkdn9" event={"ID":"4d601642-0fa6-4b68-b9cf-324c59e41637","Type":"ContainerStarted","Data":"6ea143daa8dd86582cfac8678785e557cfa411d4f6db93c56983919ad2d411bc"} Nov 02 17:29:49.499137 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:49.498798 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bkdn9" Nov 02 17:29:49.500586 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:49.500555 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fdtgx" event={"ID":"5cb013c0-4887-488b-867f-841fb6b01ca5","Type":"ContainerStarted","Data":"71f6e3c6cbe8167f959d411ab5c77b008464c0502f5365b46b8caed43dd208b2"} Nov 02 17:29:49.500692 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:49.500609 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fdtgx" event={"ID":"5cb013c0-4887-488b-867f-841fb6b01ca5","Type":"ContainerStarted","Data":"e6c91cedb10b7a9f0b90fe261cec4ad3e9bad0a388887e78d167e92ea13ef8c2"} Nov 02 17:29:50.731967 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:50.731934 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:29:50.732354 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:50.732136 4471 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 17:29:51.265251 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:51.265211 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hsdff" Nov 02 17:29:51.325445 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:51.325416 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/007d7841-411c-4b0f-b8aa-d36097b6bf31-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:51.325621 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:29:51.325571 4471 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 17:29:51.325690 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:29:51.325643 4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/007d7841-411c-4b0f-b8aa-d36097b6bf31-insights-runtime-extractor-tls podName:007d7841-411c-4b0f-b8aa-d36097b6bf31 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:59.32562472 +0000 UTC m=+17.020980188 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/007d7841-411c-4b0f-b8aa-d36097b6bf31-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ccwcv" (UID: "007d7841-411c-4b0f-b8aa-d36097b6bf31") : secret "insights-runtime-extractor-tls" not found Nov 02 17:29:58.636065 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:58.636034 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rt5cp" Nov 02 17:29:59.377361 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:59.377294 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/007d7841-411c-4b0f-b8aa-d36097b6bf31-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:59.379594 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:59.379568 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/007d7841-411c-4b0f-b8aa-d36097b6bf31-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ccwcv\" (UID: \"007d7841-411c-4b0f-b8aa-d36097b6bf31\") " pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:59.431805 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:59.431778 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ccwcv" Nov 02 17:29:59.506000 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:59.505971 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bkdn9" Nov 02 17:29:59.727577 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:29:59.727495 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ccwcv"] Nov 02 17:29:59.730072 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:29:59.730045 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod007d7841_411c_4b0f_b8aa_d36097b6bf31.slice/crio-2d9342928f73c3a9a068808185feb35fd819cc1adf0b5ff6d14d05096819bcdd WatchSource:0}: Error finding container 2d9342928f73c3a9a068808185feb35fd819cc1adf0b5ff6d14d05096819bcdd: Status 404 returned error can't find the container with id 2d9342928f73c3a9a068808185feb35fd819cc1adf0b5ff6d14d05096819bcdd Nov 02 17:30:00.537521 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:00.537438 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ccwcv" event={"ID":"007d7841-411c-4b0f-b8aa-d36097b6bf31","Type":"ContainerStarted","Data":"58f05f366830865967c1471cf94646ede44b6a9319a840437edf6ad43f002449"} Nov 02 17:30:00.537521 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:00.537497 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ccwcv" event={"ID":"007d7841-411c-4b0f-b8aa-d36097b6bf31","Type":"ContainerStarted","Data":"b999077f4749ad7c07bc80c52a76f8623df95580b1084e26cb8292f8b4ce9fd5"} Nov 02 17:30:00.537521 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:00.537518 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ccwcv" event={"ID":"007d7841-411c-4b0f-b8aa-d36097b6bf31","Type":"ContainerStarted","Data":"2d9342928f73c3a9a068808185feb35fd819cc1adf0b5ff6d14d05096819bcdd"} Nov 02 17:30:01.542922 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:01.542891 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ccwcv" event={"ID":"007d7841-411c-4b0f-b8aa-d36097b6bf31","Type":"ContainerStarted","Data":"fbbfd0a17d930f08679dccd547d9436da518f37f9a002640bfe470be3588d273"} Nov 02 17:30:03.581261 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:03.581234 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zq8j" Nov 02 17:30:04.564987 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:04.564949 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-rcjnj"] Nov 02 17:30:04.567665 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:04.567650 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcjnj" Nov 02 17:30:04.589147 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:04.589116 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 02 17:30:04.599342 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:04.599294 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 02 17:30:04.599487 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:04.599393 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 17:30:04.602671 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:04.602654 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 17:30:04.602671 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:04.602655 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 17:30:04.612183 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:04.612164 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-zz5mb\"" Nov 02 17:30:04.613195 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:04.613177 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-rcjnj"] Nov 02 17:30:04.713893 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:04.713862 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1cfc0b3c-a7f2-4cae-99ba-23286bfb0cc1-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rcjnj\" (UID: \"1cfc0b3c-a7f2-4cae-99ba-23286bfb0cc1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcjnj" Nov 02 17:30:04.714068 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:04.713907 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1cfc0b3c-a7f2-4cae-99ba-23286bfb0cc1-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rcjnj\" (UID: \"1cfc0b3c-a7f2-4cae-99ba-23286bfb0cc1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcjnj" Nov 02 17:30:04.714068 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:04.713939 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1cfc0b3c-a7f2-4cae-99ba-23286bfb0cc1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rcjnj\" (UID: \"1cfc0b3c-a7f2-4cae-99ba-23286bfb0cc1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcjnj" Nov 02 17:30:04.714068 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:04.713986 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjdfm\" (UniqueName: \"kubernetes.io/projected/1cfc0b3c-a7f2-4cae-99ba-23286bfb0cc1-kube-api-access-zjdfm\") pod \"prometheus-operator-9cd89b74c-rcjnj\" (UID: \"1cfc0b3c-a7f2-4cae-99ba-23286bfb0cc1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcjnj" Nov 02 17:30:04.815245 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:04.815167 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1cfc0b3c-a7f2-4cae-99ba-23286bfb0cc1-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rcjnj\" (UID: \"1cfc0b3c-a7f2-4cae-99ba-23286bfb0cc1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcjnj" Nov 02 17:30:04.815245 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:04.815205 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1cfc0b3c-a7f2-4cae-99ba-23286bfb0cc1-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rcjnj\" (UID: \"1cfc0b3c-a7f2-4cae-99ba-23286bfb0cc1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcjnj" Nov 02 17:30:04.815440 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:04.815362 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1cfc0b3c-a7f2-4cae-99ba-23286bfb0cc1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rcjnj\" (UID: \"1cfc0b3c-a7f2-4cae-99ba-23286bfb0cc1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcjnj" Nov 02 17:30:04.815440 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:04.815401 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjdfm\" (UniqueName: \"kubernetes.io/projected/1cfc0b3c-a7f2-4cae-99ba-23286bfb0cc1-kube-api-access-zjdfm\") pod \"prometheus-operator-9cd89b74c-rcjnj\" (UID: \"1cfc0b3c-a7f2-4cae-99ba-23286bfb0cc1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcjnj" Nov 02 17:30:04.816378 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:04.816352 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1cfc0b3c-a7f2-4cae-99ba-23286bfb0cc1-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rcjnj\" (UID: \"1cfc0b3c-a7f2-4cae-99ba-23286bfb0cc1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcjnj" Nov 02 17:30:04.820054 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:04.820029 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1cfc0b3c-a7f2-4cae-99ba-23286bfb0cc1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rcjnj\" (UID: \"1cfc0b3c-a7f2-4cae-99ba-23286bfb0cc1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcjnj" Nov 02 17:30:04.820232 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:04.820209 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1cfc0b3c-a7f2-4cae-99ba-23286bfb0cc1-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rcjnj\" (UID: \"1cfc0b3c-a7f2-4cae-99ba-23286bfb0cc1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcjnj" Nov 02 17:30:04.840412 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:04.840384 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjdfm\" (UniqueName: \"kubernetes.io/projected/1cfc0b3c-a7f2-4cae-99ba-23286bfb0cc1-kube-api-access-zjdfm\") pod \"prometheus-operator-9cd89b74c-rcjnj\" (UID: \"1cfc0b3c-a7f2-4cae-99ba-23286bfb0cc1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcjnj" Nov 02 17:30:04.879300 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:04.879276 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcjnj" Nov 02 17:30:05.175439 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:05.175415 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-rcjnj"] Nov 02 17:30:05.176936 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:30:05.176912 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cfc0b3c_a7f2_4cae_99ba_23286bfb0cc1.slice/crio-5ba3917e26871967b2d3cab12ce4b095f7b60c341863e4e198ce068a71b7d08a WatchSource:0}: Error finding container 5ba3917e26871967b2d3cab12ce4b095f7b60c341863e4e198ce068a71b7d08a: Status 404 returned error can't find the container with id 5ba3917e26871967b2d3cab12ce4b095f7b60c341863e4e198ce068a71b7d08a Nov 02 17:30:05.558252 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:05.558223 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcjnj" event={"ID":"1cfc0b3c-a7f2-4cae-99ba-23286bfb0cc1","Type":"ContainerStarted","Data":"5ba3917e26871967b2d3cab12ce4b095f7b60c341863e4e198ce068a71b7d08a"} Nov 02 17:30:06.562918 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:06.562892 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcjnj" event={"ID":"1cfc0b3c-a7f2-4cae-99ba-23286bfb0cc1","Type":"ContainerStarted","Data":"de77cafd86101ffb2ac7ce51d6b897d94ca8bb732d4e766eb38284de071caecf"} Nov 02 17:30:07.569184 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:07.569151 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcjnj" event={"ID":"1cfc0b3c-a7f2-4cae-99ba-23286bfb0cc1","Type":"ContainerStarted","Data":"f952f70c89a55d702bedc463deb96f5345c9d5499d3e15069dc7a85a9242a66f"} Nov 02 17:30:07.617562 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:07.617509 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcjnj" podStartSLOduration=2.298009188 podStartE2EDuration="3.617495953s" totalImagesPullingTime="1.319486765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:30:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:30:07.616464252 +0000 UTC m=+25.311819726" watchObservedRunningTime="2026-11-02 17:30:07.617495953 +0000 UTC m=+25.312851423" Nov 02 17:30:08.228658 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:08.228624 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-b8c96d88b-cnbq8"] Nov 02 17:30:08.230819 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:08.230802 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b8c96d88b-cnbq8" Nov 02 17:30:08.249993 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:08.249967 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 17:30:08.250439 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:08.250419 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 17:30:08.263141 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:08.263114 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 02 17:30:08.263242 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:08.263165 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 02 17:30:08.263331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:08.263298 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 02 17:30:08.263434 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:08.263411 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 02 17:30:08.263559 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:08.263491 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 02 17:30:08.263627 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:08.263565 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dkrmt\"" Nov 02 17:30:08.264148 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:08.264125 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b8c96d88b-cnbq8"] Nov 02 17:30:08.336013 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:08.335986 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80554576-a567-4705-acd5-2b933615e02d-console-serving-cert\") pod \"console-b8c96d88b-cnbq8\" (UID: \"80554576-a567-4705-acd5-2b933615e02d\") " pod="openshift-console/console-b8c96d88b-cnbq8" Nov 02 17:30:08.336173 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:08.336021 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt777\" (UniqueName: \"kubernetes.io/projected/80554576-a567-4705-acd5-2b933615e02d-kube-api-access-qt777\") pod \"console-b8c96d88b-cnbq8\" (UID: \"80554576-a567-4705-acd5-2b933615e02d\") " pod="openshift-console/console-b8c96d88b-cnbq8" Nov 02 17:30:08.336173 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:08.336051 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80554576-a567-4705-acd5-2b933615e02d-oauth-serving-cert\") pod \"console-b8c96d88b-cnbq8\" (UID: \"80554576-a567-4705-acd5-2b933615e02d\") " pod="openshift-console/console-b8c96d88b-cnbq8" Nov 02 17:30:08.336243 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:08.336170 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80554576-a567-4705-acd5-2b933615e02d-console-config\") pod \"console-b8c96d88b-cnbq8\" (UID: \"80554576-a567-4705-acd5-2b933615e02d\") " pod="openshift-console/console-b8c96d88b-cnbq8" Nov 02 17:30:08.336243 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:08.336198 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80554576-a567-4705-acd5-2b933615e02d-service-ca\") pod \"console-b8c96d88b-cnbq8\" (UID: \"80554576-a567-4705-acd5-2b933615e02d\") " pod="openshift-console/console-b8c96d88b-cnbq8" Nov 02 17:30:08.336243 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:08.336225 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80554576-a567-4705-acd5-2b933615e02d-console-oauth-config\") pod \"console-b8c96d88b-cnbq8\" (UID: \"80554576-a567-4705-acd5-2b933615e02d\") " pod="openshift-console/console-b8c96d88b-cnbq8" Nov 02 17:30:08.336362 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:08.336298 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80554576-a567-4705-acd5-2b933615e02d-tmp\") pod \"console-b8c96d88b-cnbq8\" (UID: \"80554576-a567-4705-acd5-2b933615e02d\") " pod="openshift-console/console-b8c96d88b-cnbq8" Nov 02 17:30:08.436657 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:08.436628 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80554576-a567-4705-acd5-2b933615e02d-oauth-serving-cert\") pod \"console-b8c96d88b-cnbq8\" (UID: \"80554576-a567-4705-acd5-2b933615e02d\") " pod="openshift-console/console-b8c96d88b-cnbq8" Nov 02 17:30:08.436842 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:08.436665 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80554576-a567-4705-acd5-2b933615e02d-console-config\") pod \"console-b8c96d88b-cnbq8\" (UID: \"80554576-a567-4705-acd5-2b933615e02d\") " pod="openshift-console/console-b8c96d88b-cnbq8" Nov 02 17:30:08.436842 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:08.436681 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80554576-a567-4705-acd5-2b933615e02d-service-ca\") pod \"console-b8c96d88b-cnbq8\" (UID: \"80554576-a567-4705-acd5-2b933615e02d\") " pod="openshift-console/console-b8c96d88b-cnbq8" Nov 02 17:30:08.436842 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:08.436698 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80554576-a567-4705-acd5-2b933615e02d-console-oauth-config\") pod \"console-b8c96d88b-cnbq8\" (UID: \"80554576-a567-4705-acd5-2b933615e02d\") " pod="openshift-console/console-b8c96d88b-cnbq8" Nov 02 17:30:08.436842 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:08.436716 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80554576-a567-4705-acd5-2b933615e02d-tmp\") pod \"console-b8c96d88b-cnbq8\" (UID: \"80554576-a567-4705-acd5-2b933615e02d\") " pod="openshift-console/console-b8c96d88b-cnbq8" Nov 02 17:30:08.436842 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:08.436745 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80554576-a567-4705-acd5-2b933615e02d-console-serving-cert\") pod \"console-b8c96d88b-cnbq8\" (UID: \"80554576-a567-4705-acd5-2b933615e02d\") " pod="openshift-console/console-b8c96d88b-cnbq8" Nov 02 17:30:08.437466 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:08.437123 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt777\" (UniqueName: \"kubernetes.io/projected/80554576-a567-4705-acd5-2b933615e02d-kube-api-access-qt777\") pod \"console-b8c96d88b-cnbq8\" (UID: \"80554576-a567-4705-acd5-2b933615e02d\") " pod="openshift-console/console-b8c96d88b-cnbq8" Nov 02 17:30:08.437466 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:08.437207 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80554576-a567-4705-acd5-2b933615e02d-tmp\") pod \"console-b8c96d88b-cnbq8\" (UID: \"80554576-a567-4705-acd5-2b933615e02d\") " pod="openshift-console/console-b8c96d88b-cnbq8" Nov 02 17:30:08.437627 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:08.437520 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80554576-a567-4705-acd5-2b933615e02d-oauth-serving-cert\") pod \"console-b8c96d88b-cnbq8\" (UID: \"80554576-a567-4705-acd5-2b933615e02d\") " pod="openshift-console/console-b8c96d88b-cnbq8" Nov 02 17:30:08.437627 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:08.437532 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80554576-a567-4705-acd5-2b933615e02d-console-config\") pod \"console-b8c96d88b-cnbq8\" (UID: \"80554576-a567-4705-acd5-2b933615e02d\") " pod="openshift-console/console-b8c96d88b-cnbq8" Nov 02 17:30:08.437627 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:08.437546 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80554576-a567-4705-acd5-2b933615e02d-service-ca\") pod \"console-b8c96d88b-cnbq8\" (UID: \"80554576-a567-4705-acd5-2b933615e02d\") " pod="openshift-console/console-b8c96d88b-cnbq8" Nov 02 17:30:08.439176 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:08.439152 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80554576-a567-4705-acd5-2b933615e02d-console-serving-cert\") pod \"console-b8c96d88b-cnbq8\" (UID: \"80554576-a567-4705-acd5-2b933615e02d\") " pod="openshift-console/console-b8c96d88b-cnbq8" Nov 02 17:30:08.439267 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:08.439206 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80554576-a567-4705-acd5-2b933615e02d-console-oauth-config\") pod \"console-b8c96d88b-cnbq8\" (UID: \"80554576-a567-4705-acd5-2b933615e02d\") " pod="openshift-console/console-b8c96d88b-cnbq8" Nov 02 17:30:08.460914 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:08.460887 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt777\" (UniqueName: \"kubernetes.io/projected/80554576-a567-4705-acd5-2b933615e02d-kube-api-access-qt777\") pod \"console-b8c96d88b-cnbq8\" (UID: \"80554576-a567-4705-acd5-2b933615e02d\") " pod="openshift-console/console-b8c96d88b-cnbq8" Nov 02 17:30:08.541672 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:08.541646 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b8c96d88b-cnbq8" Nov 02 17:30:08.832368 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:08.832343 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b8c96d88b-cnbq8"] Nov 02 17:30:08.834002 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:30:08.833975 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80554576_a567_4705_acd5_2b933615e02d.slice/crio-a9c96c06d13fe39812a1b76d3d72204dec73e4a2e0bf15cd73245667fc39758a WatchSource:0}: Error finding container a9c96c06d13fe39812a1b76d3d72204dec73e4a2e0bf15cd73245667fc39758a: Status 404 returned error can't find the container with id a9c96c06d13fe39812a1b76d3d72204dec73e4a2e0bf15cd73245667fc39758a Nov 02 17:30:09.181677 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:30:09.181601 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:30:09.578807 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:09.578770 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b8c96d88b-cnbq8" event={"ID":"80554576-a567-4705-acd5-2b933615e02d","Type":"ContainerStarted","Data":"a9c96c06d13fe39812a1b76d3d72204dec73e4a2e0bf15cd73245667fc39758a"} Nov 02 17:30:10.554973 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.554935 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-v5w55"] Nov 02 17:30:10.557536 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.557513 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v5w55" Nov 02 17:30:10.577275 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.577252 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-6hfdn"] Nov 02 17:30:10.579663 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.579642 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6hfdn" Nov 02 17:30:10.589380 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.589357 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 17:30:10.589471 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.589380 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 17:30:10.589471 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.589419 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sn4p8\"" Nov 02 17:30:10.589563 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.589474 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 17:30:10.628757 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.628720 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-6hfdn"] Nov 02 17:30:10.679499 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.679468 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 17:30:10.683673 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.683629 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 17:30:10.683800 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.683722 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 17:30:10.683800 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.683738 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-lkwsj\"" Nov 02 17:30:10.753325 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.753274 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/219e320d-1e5e-4c4f-9eaf-5ee7411104b0-node-exporter-tls\") pod \"node-exporter-v5w55\" (UID: \"219e320d-1e5e-4c4f-9eaf-5ee7411104b0\") " pod="openshift-monitoring/node-exporter-v5w55" Nov 02 17:30:10.753493 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.753338 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/219e320d-1e5e-4c4f-9eaf-5ee7411104b0-node-exporter-accelerators-collector-config\") pod \"node-exporter-v5w55\" (UID: \"219e320d-1e5e-4c4f-9eaf-5ee7411104b0\") " pod="openshift-monitoring/node-exporter-v5w55" Nov 02 17:30:10.753493 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.753361 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-6hfdn\" (UID: \"6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6hfdn" Nov 02 17:30:10.753493 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.753384 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/219e320d-1e5e-4c4f-9eaf-5ee7411104b0-node-exporter-textfile\") pod \"node-exporter-v5w55\" (UID: \"219e320d-1e5e-4c4f-9eaf-5ee7411104b0\") " pod="openshift-monitoring/node-exporter-v5w55" Nov 02 17:30:10.753493 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.753413 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/219e320d-1e5e-4c4f-9eaf-5ee7411104b0-root\") pod \"node-exporter-v5w55\" (UID: \"219e320d-1e5e-4c4f-9eaf-5ee7411104b0\") " pod="openshift-monitoring/node-exporter-v5w55" Nov 02 17:30:10.753667 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.753577 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-6hfdn\" (UID: \"6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6hfdn" Nov 02 17:30:10.753667 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.753619 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/219e320d-1e5e-4c4f-9eaf-5ee7411104b0-node-exporter-wtmp\") pod \"node-exporter-v5w55\" (UID: \"219e320d-1e5e-4c4f-9eaf-5ee7411104b0\") " pod="openshift-monitoring/node-exporter-v5w55" Nov 02 17:30:10.753667 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.753642 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/219e320d-1e5e-4c4f-9eaf-5ee7411104b0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v5w55\" (UID: \"219e320d-1e5e-4c4f-9eaf-5ee7411104b0\") " pod="openshift-monitoring/node-exporter-v5w55" Nov 02 17:30:10.753809 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.753702 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-6hfdn\" (UID: \"6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6hfdn" Nov 02 17:30:10.753809 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.753780 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-6hfdn\" (UID: \"6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6hfdn" Nov 02 17:30:10.753895 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.753810 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrqw2\" (UniqueName: \"kubernetes.io/projected/6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c-kube-api-access-rrqw2\") pod \"kube-state-metrics-5dc8d7d89b-6hfdn\" (UID: \"6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6hfdn" Nov 02 17:30:10.753895 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.753860 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/219e320d-1e5e-4c4f-9eaf-5ee7411104b0-metrics-client-ca\") pod \"node-exporter-v5w55\" (UID: \"219e320d-1e5e-4c4f-9eaf-5ee7411104b0\") " pod="openshift-monitoring/node-exporter-v5w55" Nov 02 17:30:10.753966 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.753952 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/219e320d-1e5e-4c4f-9eaf-5ee7411104b0-sys\") pod \"node-exporter-v5w55\" (UID: \"219e320d-1e5e-4c4f-9eaf-5ee7411104b0\") " pod="openshift-monitoring/node-exporter-v5w55" Nov 02 17:30:10.754017 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.753980 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmvd2\" (UniqueName: \"kubernetes.io/projected/219e320d-1e5e-4c4f-9eaf-5ee7411104b0-kube-api-access-pmvd2\") pod \"node-exporter-v5w55\" (UID: \"219e320d-1e5e-4c4f-9eaf-5ee7411104b0\") " pod="openshift-monitoring/node-exporter-v5w55" Nov 02 17:30:10.754110 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.754091 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-6hfdn\" (UID: \"6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6hfdn" Nov 02 17:30:10.855296 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.855220 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/219e320d-1e5e-4c4f-9eaf-5ee7411104b0-root\") pod \"node-exporter-v5w55\" (UID: \"219e320d-1e5e-4c4f-9eaf-5ee7411104b0\") " pod="openshift-monitoring/node-exporter-v5w55" Nov 02 17:30:10.855296 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.855270 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-6hfdn\" (UID: \"6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6hfdn" Nov 02 17:30:10.855528 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.855296 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/219e320d-1e5e-4c4f-9eaf-5ee7411104b0-node-exporter-wtmp\") pod \"node-exporter-v5w55\" (UID: \"219e320d-1e5e-4c4f-9eaf-5ee7411104b0\") " pod="openshift-monitoring/node-exporter-v5w55" Nov 02 17:30:10.855528 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.855339 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/219e320d-1e5e-4c4f-9eaf-5ee7411104b0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v5w55\" (UID: \"219e320d-1e5e-4c4f-9eaf-5ee7411104b0\") " pod="openshift-monitoring/node-exporter-v5w55" Nov 02 17:30:10.855528 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.855359 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/219e320d-1e5e-4c4f-9eaf-5ee7411104b0-root\") pod \"node-exporter-v5w55\" (UID: \"219e320d-1e5e-4c4f-9eaf-5ee7411104b0\") " pod="openshift-monitoring/node-exporter-v5w55" Nov 02 17:30:10.855528 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.855364 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-6hfdn\" (UID: \"6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6hfdn" Nov 02 17:30:10.855528 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.855433 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-6hfdn\" (UID: \"6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6hfdn" Nov 02 17:30:10.855528 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.855454 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/219e320d-1e5e-4c4f-9eaf-5ee7411104b0-node-exporter-wtmp\") pod \"node-exporter-v5w55\" (UID: \"219e320d-1e5e-4c4f-9eaf-5ee7411104b0\") " pod="openshift-monitoring/node-exporter-v5w55" Nov 02 17:30:10.855528 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.855460 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrqw2\" (UniqueName: \"kubernetes.io/projected/6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c-kube-api-access-rrqw2\") pod \"kube-state-metrics-5dc8d7d89b-6hfdn\" (UID: \"6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6hfdn" Nov 02 17:30:10.855528 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.855503 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/219e320d-1e5e-4c4f-9eaf-5ee7411104b0-metrics-client-ca\") pod \"node-exporter-v5w55\" (UID: \"219e320d-1e5e-4c4f-9eaf-5ee7411104b0\") " pod="openshift-monitoring/node-exporter-v5w55" Nov 02 17:30:10.855917 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.855541 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/219e320d-1e5e-4c4f-9eaf-5ee7411104b0-sys\") pod \"node-exporter-v5w55\" (UID: \"219e320d-1e5e-4c4f-9eaf-5ee7411104b0\") " pod="openshift-monitoring/node-exporter-v5w55" Nov 02 17:30:10.855917 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.855567 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmvd2\" (UniqueName: \"kubernetes.io/projected/219e320d-1e5e-4c4f-9eaf-5ee7411104b0-kube-api-access-pmvd2\") pod \"node-exporter-v5w55\" (UID: \"219e320d-1e5e-4c4f-9eaf-5ee7411104b0\") " pod="openshift-monitoring/node-exporter-v5w55" Nov 02 17:30:10.855917 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.855613 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-6hfdn\" (UID: \"6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6hfdn" Nov 02 17:30:10.855917 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.855653 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/219e320d-1e5e-4c4f-9eaf-5ee7411104b0-node-exporter-tls\") pod \"node-exporter-v5w55\" (UID: \"219e320d-1e5e-4c4f-9eaf-5ee7411104b0\") " pod="openshift-monitoring/node-exporter-v5w55" Nov 02 17:30:10.855917 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.855690 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/219e320d-1e5e-4c4f-9eaf-5ee7411104b0-node-exporter-accelerators-collector-config\") pod \"node-exporter-v5w55\" (UID: \"219e320d-1e5e-4c4f-9eaf-5ee7411104b0\") " pod="openshift-monitoring/node-exporter-v5w55" Nov 02 17:30:10.855917 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.855720 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-6hfdn\" (UID: \"6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6hfdn" Nov 02 17:30:10.855917 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.855758 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/219e320d-1e5e-4c4f-9eaf-5ee7411104b0-node-exporter-textfile\") pod \"node-exporter-v5w55\" (UID: \"219e320d-1e5e-4c4f-9eaf-5ee7411104b0\") " pod="openshift-monitoring/node-exporter-v5w55" Nov 02 17:30:10.856296 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.856106 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-6hfdn\" (UID: \"6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6hfdn" Nov 02 17:30:10.856758 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.856455 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-6hfdn\" (UID: \"6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6hfdn" Nov 02 17:30:10.856758 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.856486 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/219e320d-1e5e-4c4f-9eaf-5ee7411104b0-metrics-client-ca\") pod \"node-exporter-v5w55\" (UID: \"219e320d-1e5e-4c4f-9eaf-5ee7411104b0\") " pod="openshift-monitoring/node-exporter-v5w55" Nov 02 17:30:10.856758 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.856523 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/219e320d-1e5e-4c4f-9eaf-5ee7411104b0-sys\") pod \"node-exporter-v5w55\" (UID: \"219e320d-1e5e-4c4f-9eaf-5ee7411104b0\") " pod="openshift-monitoring/node-exporter-v5w55" Nov 02 17:30:10.856758 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.856617 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/219e320d-1e5e-4c4f-9eaf-5ee7411104b0-node-exporter-accelerators-collector-config\") pod \"node-exporter-v5w55\" (UID: \"219e320d-1e5e-4c4f-9eaf-5ee7411104b0\") " pod="openshift-monitoring/node-exporter-v5w55" Nov 02 17:30:10.856758 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.856728 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/219e320d-1e5e-4c4f-9eaf-5ee7411104b0-node-exporter-textfile\") pod \"node-exporter-v5w55\" (UID: \"219e320d-1e5e-4c4f-9eaf-5ee7411104b0\") " pod="openshift-monitoring/node-exporter-v5w55" Nov 02 17:30:10.857042 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.856839 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-6hfdn\" (UID: \"6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6hfdn" Nov 02 17:30:10.858618 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.858569 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-6hfdn\" (UID: \"6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6hfdn" Nov 02 17:30:10.858743 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.858725 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/219e320d-1e5e-4c4f-9eaf-5ee7411104b0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v5w55\" (UID: \"219e320d-1e5e-4c4f-9eaf-5ee7411104b0\") " pod="openshift-monitoring/node-exporter-v5w55" Nov 02 17:30:10.858918 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.858899 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-6hfdn\" (UID: \"6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6hfdn" Nov 02 17:30:10.861667 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.861646 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/219e320d-1e5e-4c4f-9eaf-5ee7411104b0-node-exporter-tls\") pod \"node-exporter-v5w55\" (UID: \"219e320d-1e5e-4c4f-9eaf-5ee7411104b0\") " pod="openshift-monitoring/node-exporter-v5w55" Nov 02 17:30:10.880284 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.880251 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrqw2\" (UniqueName: \"kubernetes.io/projected/6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c-kube-api-access-rrqw2\") pod \"kube-state-metrics-5dc8d7d89b-6hfdn\" (UID: \"6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6hfdn" Nov 02 17:30:10.887613 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.887566 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmvd2\" (UniqueName: \"kubernetes.io/projected/219e320d-1e5e-4c4f-9eaf-5ee7411104b0-kube-api-access-pmvd2\") pod \"node-exporter-v5w55\" (UID: \"219e320d-1e5e-4c4f-9eaf-5ee7411104b0\") " pod="openshift-monitoring/node-exporter-v5w55" Nov 02 17:30:10.893416 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:10.893393 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6hfdn" Nov 02 17:30:11.172427 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.172351 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v5w55" Nov 02 17:30:11.187198 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:30:11.187166 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod219e320d_1e5e_4c4f_9eaf_5ee7411104b0.slice/crio-0d5a87c81b878719b8fe818927dcd6714bdc13116d0016367d77fc9a5b098dfb WatchSource:0}: Error finding container 0d5a87c81b878719b8fe818927dcd6714bdc13116d0016367d77fc9a5b098dfb: Status 404 returned error can't find the container with id 0d5a87c81b878719b8fe818927dcd6714bdc13116d0016367d77fc9a5b098dfb Nov 02 17:30:11.221058 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.221030 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-6hfdn"] Nov 02 17:30:11.222810 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:30:11.222784 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fd6596b_f5c4_4ad5_b5d8_79be4df1b56c.slice/crio-bb788b1f19bf07c6769fe4f304653892d892add603e9a467ed905adb8f227eab WatchSource:0}: Error finding container bb788b1f19bf07c6769fe4f304653892d892add603e9a467ed905adb8f227eab: Status 404 returned error can't find the container with id bb788b1f19bf07c6769fe4f304653892d892add603e9a467ed905adb8f227eab Nov 02 17:30:11.268722 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.268696 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hsdff" Nov 02 17:30:11.587272 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.587230 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v5w55" event={"ID":"219e320d-1e5e-4c4f-9eaf-5ee7411104b0","Type":"ContainerStarted","Data":"0d5a87c81b878719b8fe818927dcd6714bdc13116d0016367d77fc9a5b098dfb"} Nov 02 17:30:11.589406 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.589372 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b8c96d88b-cnbq8" event={"ID":"80554576-a567-4705-acd5-2b933615e02d","Type":"ContainerStarted","Data":"7adf2f7ce4a21b314c5f3c817afd36573102cc8850d49796eb6ee835f3088b6c"} Nov 02 17:30:11.590751 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.590723 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6hfdn" event={"ID":"6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c","Type":"ContainerStarted","Data":"bb788b1f19bf07c6769fe4f304653892d892add603e9a467ed905adb8f227eab"} Nov 02 17:30:11.722407 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.721639 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-b8c96d88b-cnbq8" podStartSLOduration=1.175779393 podStartE2EDuration="3.72161884s" totalImagesPullingTime="2.545839447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:30:11.637795736 +0000 UTC m=+29.333151239" watchObservedRunningTime="2026-11-02 17:30:11.72161884 +0000 UTC m=+29.416974315" Nov 02 17:30:11.722407 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.722148 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:30:11.729829 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.729802 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.776924 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.776896 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 17:30:11.777114 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.777096 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 17:30:11.777325 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.777287 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-h948w\"" Nov 02 17:30:11.777432 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.777378 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 17:30:11.777580 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.777565 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 17:30:11.777692 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.777678 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 17:30:11.777757 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.777748 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 17:30:11.777823 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.777813 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 17:30:11.777945 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.777885 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 17:30:11.777998 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.777954 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 17:30:11.778216 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.778196 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:30:11.866774 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.866737 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-web-config\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.866945 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.866859 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3111c93-5a21-4a5b-8ada-e78c73ebf841-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.866945 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.866899 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d3111c93-5a21-4a5b-8ada-e78c73ebf841-config-out\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.866945 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.866929 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.867118 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.866955 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8ptt\" (UniqueName: \"kubernetes.io/projected/d3111c93-5a21-4a5b-8ada-e78c73ebf841-kube-api-access-k8ptt\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.867118 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.866991 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.867118 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.867022 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d3111c93-5a21-4a5b-8ada-e78c73ebf841-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.867118 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.867064 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3111c93-5a21-4a5b-8ada-e78c73ebf841-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.867118 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.867101 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-config-volume\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.867374 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.867134 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.867374 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.867165 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.867374 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.867189 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.867374 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.867217 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d3111c93-5a21-4a5b-8ada-e78c73ebf841-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.968154 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.968120 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.968370 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.968164 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.968370 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.968190 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.968370 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.968213 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d3111c93-5a21-4a5b-8ada-e78c73ebf841-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.968370 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.968253 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-web-config\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.968370 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.968290 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3111c93-5a21-4a5b-8ada-e78c73ebf841-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.968370 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.968349 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d3111c93-5a21-4a5b-8ada-e78c73ebf841-config-out\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.968682 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.968376 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.968682 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.968409 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8ptt\" (UniqueName: \"kubernetes.io/projected/d3111c93-5a21-4a5b-8ada-e78c73ebf841-kube-api-access-k8ptt\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.968682 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.968444 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.968682 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.968488 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d3111c93-5a21-4a5b-8ada-e78c73ebf841-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.968682 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.968531 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3111c93-5a21-4a5b-8ada-e78c73ebf841-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.968682 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.968571 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-config-volume\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.969494 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.969465 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d3111c93-5a21-4a5b-8ada-e78c73ebf841-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.972227 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.971469 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.972227 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.971525 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.972227 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.971526 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d3111c93-5a21-4a5b-8ada-e78c73ebf841-config-out\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.972227 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.972134 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3111c93-5a21-4a5b-8ada-e78c73ebf841-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.974228 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.974123 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3111c93-5a21-4a5b-8ada-e78c73ebf841-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.975181 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.975155 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-config-volume\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.976362 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.976335 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-web-config\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.976700 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.976677 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.977933 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.977907 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.978732 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.978707 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d3111c93-5a21-4a5b-8ada-e78c73ebf841-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:11.978820 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:11.978794 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:12.012524 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:12.012463 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8ptt\" (UniqueName: \"kubernetes.io/projected/d3111c93-5a21-4a5b-8ada-e78c73ebf841-kube-api-access-k8ptt\") pod \"alertmanager-main-0\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:12.075479 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:12.075449 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:30:12.442688 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:12.442660 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:30:12.453845 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:30:12.453821 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3111c93_5a21_4a5b_8ada_e78c73ebf841.slice/crio-c28a4819fe5cfd14fcc3706e868411d5794be98e628a2a0f473b7c5ee212e495 WatchSource:0}: Error finding container c28a4819fe5cfd14fcc3706e868411d5794be98e628a2a0f473b7c5ee212e495: Status 404 returned error can't find the container with id c28a4819fe5cfd14fcc3706e868411d5794be98e628a2a0f473b7c5ee212e495 Nov 02 17:30:12.596480 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:12.596439 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v5w55" event={"ID":"219e320d-1e5e-4c4f-9eaf-5ee7411104b0","Type":"ContainerStarted","Data":"e84cf27058160a8c2a0e1831a5b5b813a72e38dc65f4c12b96a37cdb6065bc6a"} Nov 02 17:30:12.598167 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:12.598143 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3111c93-5a21-4a5b-8ada-e78c73ebf841","Type":"ContainerStarted","Data":"c28a4819fe5cfd14fcc3706e868411d5794be98e628a2a0f473b7c5ee212e495"} Nov 02 17:30:13.603325 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:13.603277 4471 generic.go:356] "Generic (PLEG): container finished" podID="219e320d-1e5e-4c4f-9eaf-5ee7411104b0" containerID="e84cf27058160a8c2a0e1831a5b5b813a72e38dc65f4c12b96a37cdb6065bc6a" exitCode=0 Nov 02 17:30:13.603792 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:13.603341 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v5w55" event={"ID":"219e320d-1e5e-4c4f-9eaf-5ee7411104b0","Type":"ContainerDied","Data":"e84cf27058160a8c2a0e1831a5b5b813a72e38dc65f4c12b96a37cdb6065bc6a"} Nov 02 17:30:13.605726 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:13.605704 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6hfdn" event={"ID":"6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c","Type":"ContainerStarted","Data":"ab7fca736c2b5a3e3275440005cee8588032e0eae2e7bc1c6088b749414e26b8"} Nov 02 17:30:13.605812 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:13.605749 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6hfdn" event={"ID":"6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c","Type":"ContainerStarted","Data":"d6a34a1ae069f4eb1b9275f0f46e30aca4719e61174fb99de39d36ab77cf383a"} Nov 02 17:30:13.605812 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:13.605768 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6hfdn" event={"ID":"6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c","Type":"ContainerStarted","Data":"f372a666c99efbedbe9b0a9e4ec941987e610e02f6ae20a6a9a76e9939f1bef7"} Nov 02 17:30:14.612652 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:14.612618 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v5w55" event={"ID":"219e320d-1e5e-4c4f-9eaf-5ee7411104b0","Type":"ContainerStarted","Data":"c8c7fd8d3484ed51512291b978390d7d2342d3898805e51afac5743e08d113a5"} Nov 02 17:30:14.613131 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:14.612680 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v5w55" event={"ID":"219e320d-1e5e-4c4f-9eaf-5ee7411104b0","Type":"ContainerStarted","Data":"c7f214a282e4fa0a548252f530c6a5b902dbcbe8d9ad0e1789583b422d2633e3"} Nov 02 17:30:14.614252 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:14.614227 4471 generic.go:356] "Generic (PLEG): container finished" podID="d3111c93-5a21-4a5b-8ada-e78c73ebf841" containerID="ec98c3a7e7a0200db86f3bfa67292ee33789fef2ac095c779739a53d2ee5f04c" exitCode=0 Nov 02 17:30:14.614416 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:14.614339 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3111c93-5a21-4a5b-8ada-e78c73ebf841","Type":"ContainerDied","Data":"ec98c3a7e7a0200db86f3bfa67292ee33789fef2ac095c779739a53d2ee5f04c"} Nov 02 17:30:14.660547 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:14.660502 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-v5w55" podStartSLOduration=3.325366513 podStartE2EDuration="4.660488828s" totalImagesPullingTime="1.259214452s" totalInitContainerRuntime="75.907863ms" isStatefulPod=true podCreationTimestamp="2026-11-02 17:30:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:30:14.659967343 +0000 UTC m=+32.355322815" watchObservedRunningTime="2026-11-02 17:30:14.660488828 +0000 UTC m=+32.355844324" Nov 02 17:30:14.660666 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:14.660600 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6hfdn" podStartSLOduration=3.15146265 podStartE2EDuration="4.660594949s" totalImagesPullingTime="1.509132299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:30:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:30:13.699858298 +0000 UTC m=+31.395213789" watchObservedRunningTime="2026-11-02 17:30:14.660594949 +0000 UTC m=+32.355950422" Nov 02 17:30:15.025324 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:15.025266 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-t75j5"] Nov 02 17:30:15.045835 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:15.045807 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-t75j5" Nov 02 17:30:15.049327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:15.049272 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-t75j5"] Nov 02 17:30:15.092999 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:15.092975 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f5525bdf-02c9-4225-b7a2-ae353e0ba168-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-t75j5\" (UID: \"f5525bdf-02c9-4225-b7a2-ae353e0ba168\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-t75j5" Nov 02 17:30:15.093139 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:15.093012 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4vqx\" (UniqueName: \"kubernetes.io/projected/f5525bdf-02c9-4225-b7a2-ae353e0ba168-kube-api-access-b4vqx\") pod \"monitoring-plugin-77f4f76594-t75j5\" (UID: \"f5525bdf-02c9-4225-b7a2-ae353e0ba168\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-t75j5" Nov 02 17:30:15.106428 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:15.106384 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 02 17:30:15.106428 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:15.106422 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-57fwl\"" Nov 02 17:30:15.193898 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:15.193864 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f5525bdf-02c9-4225-b7a2-ae353e0ba168-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-t75j5\" (UID: \"f5525bdf-02c9-4225-b7a2-ae353e0ba168\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-t75j5" Nov 02 17:30:15.194076 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:15.193923 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4vqx\" (UniqueName: \"kubernetes.io/projected/f5525bdf-02c9-4225-b7a2-ae353e0ba168-kube-api-access-b4vqx\") pod \"monitoring-plugin-77f4f76594-t75j5\" (UID: \"f5525bdf-02c9-4225-b7a2-ae353e0ba168\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-t75j5" Nov 02 17:30:15.196624 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:15.196599 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f5525bdf-02c9-4225-b7a2-ae353e0ba168-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-t75j5\" (UID: \"f5525bdf-02c9-4225-b7a2-ae353e0ba168\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-t75j5" Nov 02 17:30:15.222896 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:15.222869 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4vqx\" (UniqueName: \"kubernetes.io/projected/f5525bdf-02c9-4225-b7a2-ae353e0ba168-kube-api-access-b4vqx\") pod \"monitoring-plugin-77f4f76594-t75j5\" (UID: \"f5525bdf-02c9-4225-b7a2-ae353e0ba168\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-t75j5" Nov 02 17:30:15.376143 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:15.376062 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-t75j5" Nov 02 17:30:15.694718 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:15.694691 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-t75j5"] Nov 02 17:30:15.696352 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:30:15.696317 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5525bdf_02c9_4225_b7a2_ae353e0ba168.slice/crio-022534cd0a13d71b81904195d717c17b493b14517cd23f67acdf71c37e2d4fa2 WatchSource:0}: Error finding container 022534cd0a13d71b81904195d717c17b493b14517cd23f67acdf71c37e2d4fa2: Status 404 returned error can't find the container with id 022534cd0a13d71b81904195d717c17b493b14517cd23f67acdf71c37e2d4fa2 Nov 02 17:30:16.625868 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:16.625826 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3111c93-5a21-4a5b-8ada-e78c73ebf841","Type":"ContainerStarted","Data":"fbeb0e97cdc43442fb546ef16d8377f735480a8c3268b9a926718c1c59fc2a50"} Nov 02 17:30:16.627471 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:16.627415 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-t75j5" event={"ID":"f5525bdf-02c9-4225-b7a2-ae353e0ba168","Type":"ContainerStarted","Data":"022534cd0a13d71b81904195d717c17b493b14517cd23f67acdf71c37e2d4fa2"} Nov 02 17:30:17.636685 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:17.636600 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3111c93-5a21-4a5b-8ada-e78c73ebf841","Type":"ContainerStarted","Data":"e2cfb5c8e6ddf84e602e48c968759c1c22ccc6b4d68763d85a6cc13e5446222a"} Nov 02 17:30:17.636685 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:17.636679 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3111c93-5a21-4a5b-8ada-e78c73ebf841","Type":"ContainerStarted","Data":"9076d775b7b220e47194f834f12a2320e0a8c22f295fd89ae3d3ede3b0cea5a3"} Nov 02 17:30:17.637206 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:17.636717 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3111c93-5a21-4a5b-8ada-e78c73ebf841","Type":"ContainerStarted","Data":"dc5fc6b01424704ed71b782ced304597b143c6580d98deb33cf9f62ab759a1a9"} Nov 02 17:30:17.637206 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:17.636744 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3111c93-5a21-4a5b-8ada-e78c73ebf841","Type":"ContainerStarted","Data":"cd8c35bc4569aaee407d2177089b46ae0ecc260ec82d4a3f4797e7c23d05ee23"} Nov 02 17:30:17.638578 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:17.638544 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-t75j5" event={"ID":"f5525bdf-02c9-4225-b7a2-ae353e0ba168","Type":"ContainerStarted","Data":"0a9aa3c9faab8f1758ba300b01acd1d2372190aadac6fffdfd1393ebc76ed75f"} Nov 02 17:30:17.640986 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:17.640961 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-t75j5" Nov 02 17:30:17.647417 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:17.645686 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-t75j5" Nov 02 17:30:17.746401 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:17.746357 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-t75j5" podStartSLOduration=1.081011075 podStartE2EDuration="2.746342157s" totalImagesPullingTime="1.665331082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:30:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:30:17.745224019 +0000 UTC m=+35.440579482" watchObservedRunningTime="2026-11-02 17:30:17.746342157 +0000 UTC m=+35.441697629" Nov 02 17:30:18.542289 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:18.542260 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b8c96d88b-cnbq8" Nov 02 17:30:18.542553 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:18.542330 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b8c96d88b-cnbq8" Nov 02 17:30:18.546848 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:18.546824 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b8c96d88b-cnbq8" Nov 02 17:30:18.645516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:18.645478 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3111c93-5a21-4a5b-8ada-e78c73ebf841","Type":"ContainerStarted","Data":"8fde5fc45f9ee160bff43facaeb8938828214f7b8ef8349626297ff8699cc58b"} Nov 02 17:30:18.649290 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:18.649259 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b8c96d88b-cnbq8" Nov 02 17:30:18.715164 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:18.715107 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.693340441 podStartE2EDuration="7.715094918s" totalImagesPullingTime="4.008682807s" totalInitContainerRuntime="13.07167ms" isStatefulPod=false podCreationTimestamp="2026-11-02 17:30:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:30:18.714234089 +0000 UTC m=+36.409589553" watchObservedRunningTime="2026-11-02 17:30:18.715094918 +0000 UTC m=+36.410450390" Nov 02 17:30:20.240135 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:20.240098 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f987684bb-8llbv"] Nov 02 17:30:20.242397 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:20.242380 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f987684bb-8llbv" Nov 02 17:30:20.299403 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:20.299370 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 02 17:30:20.300415 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:20.300401 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f987684bb-8llbv"] Nov 02 17:30:20.333962 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:20.333934 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d7d1860-d847-410f-8e0e-3e5065167ed2-tmp\") pod \"console-7f987684bb-8llbv\" (UID: \"0d7d1860-d847-410f-8e0e-3e5065167ed2\") " pod="openshift-console/console-7f987684bb-8llbv" Nov 02 17:30:20.334058 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:20.333962 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzqqb\" (UniqueName: \"kubernetes.io/projected/0d7d1860-d847-410f-8e0e-3e5065167ed2-kube-api-access-hzqqb\") pod \"console-7f987684bb-8llbv\" (UID: \"0d7d1860-d847-410f-8e0e-3e5065167ed2\") " pod="openshift-console/console-7f987684bb-8llbv" Nov 02 17:30:20.334058 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:20.333986 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d7d1860-d847-410f-8e0e-3e5065167ed2-service-ca\") pod \"console-7f987684bb-8llbv\" (UID: \"0d7d1860-d847-410f-8e0e-3e5065167ed2\") " pod="openshift-console/console-7f987684bb-8llbv" Nov 02 17:30:20.334136 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:20.334118 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d7d1860-d847-410f-8e0e-3e5065167ed2-trusted-ca-bundle\") pod \"console-7f987684bb-8llbv\" (UID: \"0d7d1860-d847-410f-8e0e-3e5065167ed2\") " pod="openshift-console/console-7f987684bb-8llbv" Nov 02 17:30:20.334171 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:20.334151 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d7d1860-d847-410f-8e0e-3e5065167ed2-console-serving-cert\") pod \"console-7f987684bb-8llbv\" (UID: \"0d7d1860-d847-410f-8e0e-3e5065167ed2\") " pod="openshift-console/console-7f987684bb-8llbv" Nov 02 17:30:20.334231 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:20.334217 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d7d1860-d847-410f-8e0e-3e5065167ed2-console-config\") pod \"console-7f987684bb-8llbv\" (UID: \"0d7d1860-d847-410f-8e0e-3e5065167ed2\") " pod="openshift-console/console-7f987684bb-8llbv" Nov 02 17:30:20.334272 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:20.334258 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d7d1860-d847-410f-8e0e-3e5065167ed2-console-oauth-config\") pod \"console-7f987684bb-8llbv\" (UID: \"0d7d1860-d847-410f-8e0e-3e5065167ed2\") " pod="openshift-console/console-7f987684bb-8llbv" Nov 02 17:30:20.334335 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:20.334320 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d7d1860-d847-410f-8e0e-3e5065167ed2-oauth-serving-cert\") pod \"console-7f987684bb-8llbv\" (UID: \"0d7d1860-d847-410f-8e0e-3e5065167ed2\") " pod="openshift-console/console-7f987684bb-8llbv" Nov 02 17:30:20.435077 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:20.435049 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d7d1860-d847-410f-8e0e-3e5065167ed2-console-serving-cert\") pod \"console-7f987684bb-8llbv\" (UID: \"0d7d1860-d847-410f-8e0e-3e5065167ed2\") " pod="openshift-console/console-7f987684bb-8llbv" Nov 02 17:30:20.435221 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:20.435088 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d7d1860-d847-410f-8e0e-3e5065167ed2-console-config\") pod \"console-7f987684bb-8llbv\" (UID: \"0d7d1860-d847-410f-8e0e-3e5065167ed2\") " pod="openshift-console/console-7f987684bb-8llbv" Nov 02 17:30:20.435221 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:20.435119 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d7d1860-d847-410f-8e0e-3e5065167ed2-console-oauth-config\") pod \"console-7f987684bb-8llbv\" (UID: \"0d7d1860-d847-410f-8e0e-3e5065167ed2\") " pod="openshift-console/console-7f987684bb-8llbv" Nov 02 17:30:20.435221 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:20.435150 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d7d1860-d847-410f-8e0e-3e5065167ed2-oauth-serving-cert\") pod \"console-7f987684bb-8llbv\" (UID: \"0d7d1860-d847-410f-8e0e-3e5065167ed2\") " pod="openshift-console/console-7f987684bb-8llbv" Nov 02 17:30:20.435221 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:20.435192 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d7d1860-d847-410f-8e0e-3e5065167ed2-tmp\") pod \"console-7f987684bb-8llbv\" (UID: \"0d7d1860-d847-410f-8e0e-3e5065167ed2\") " pod="openshift-console/console-7f987684bb-8llbv" Nov 02 17:30:20.435221 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:20.435215 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzqqb\" (UniqueName: \"kubernetes.io/projected/0d7d1860-d847-410f-8e0e-3e5065167ed2-kube-api-access-hzqqb\") pod \"console-7f987684bb-8llbv\" (UID: \"0d7d1860-d847-410f-8e0e-3e5065167ed2\") " pod="openshift-console/console-7f987684bb-8llbv" Nov 02 17:30:20.435507 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:20.435251 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d7d1860-d847-410f-8e0e-3e5065167ed2-service-ca\") pod \"console-7f987684bb-8llbv\" (UID: \"0d7d1860-d847-410f-8e0e-3e5065167ed2\") " pod="openshift-console/console-7f987684bb-8llbv" Nov 02 17:30:20.435507 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:20.435322 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d7d1860-d847-410f-8e0e-3e5065167ed2-trusted-ca-bundle\") pod \"console-7f987684bb-8llbv\" (UID: \"0d7d1860-d847-410f-8e0e-3e5065167ed2\") " pod="openshift-console/console-7f987684bb-8llbv" Nov 02 17:30:20.435710 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:20.435679 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d7d1860-d847-410f-8e0e-3e5065167ed2-tmp\") pod \"console-7f987684bb-8llbv\" (UID: \"0d7d1860-d847-410f-8e0e-3e5065167ed2\") " pod="openshift-console/console-7f987684bb-8llbv" Nov 02 17:30:20.436079 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:20.436055 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d7d1860-d847-410f-8e0e-3e5065167ed2-console-config\") pod \"console-7f987684bb-8llbv\" (UID: \"0d7d1860-d847-410f-8e0e-3e5065167ed2\") " pod="openshift-console/console-7f987684bb-8llbv" Nov 02 17:30:20.436563 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:20.436521 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d7d1860-d847-410f-8e0e-3e5065167ed2-oauth-serving-cert\") pod \"console-7f987684bb-8llbv\" (UID: \"0d7d1860-d847-410f-8e0e-3e5065167ed2\") " pod="openshift-console/console-7f987684bb-8llbv" Nov 02 17:30:20.436671 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:20.436607 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d7d1860-d847-410f-8e0e-3e5065167ed2-service-ca\") pod \"console-7f987684bb-8llbv\" (UID: \"0d7d1860-d847-410f-8e0e-3e5065167ed2\") " pod="openshift-console/console-7f987684bb-8llbv" Nov 02 17:30:20.436791 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:20.436770 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d7d1860-d847-410f-8e0e-3e5065167ed2-trusted-ca-bundle\") pod \"console-7f987684bb-8llbv\" (UID: \"0d7d1860-d847-410f-8e0e-3e5065167ed2\") " pod="openshift-console/console-7f987684bb-8llbv" Nov 02 17:30:20.437901 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:20.437881 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d7d1860-d847-410f-8e0e-3e5065167ed2-console-oauth-config\") pod \"console-7f987684bb-8llbv\" (UID: \"0d7d1860-d847-410f-8e0e-3e5065167ed2\") " pod="openshift-console/console-7f987684bb-8llbv" Nov 02 17:30:20.440666 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:20.440643 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d7d1860-d847-410f-8e0e-3e5065167ed2-console-serving-cert\") pod \"console-7f987684bb-8llbv\" (UID: \"0d7d1860-d847-410f-8e0e-3e5065167ed2\") " pod="openshift-console/console-7f987684bb-8llbv" Nov 02 17:30:20.459796 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:20.459769 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzqqb\" (UniqueName: \"kubernetes.io/projected/0d7d1860-d847-410f-8e0e-3e5065167ed2-kube-api-access-hzqqb\") pod \"console-7f987684bb-8llbv\" (UID: \"0d7d1860-d847-410f-8e0e-3e5065167ed2\") " pod="openshift-console/console-7f987684bb-8llbv" Nov 02 17:30:20.552622 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:20.552598 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f987684bb-8llbv" Nov 02 17:30:20.846338 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:20.846286 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f987684bb-8llbv"] Nov 02 17:30:20.846924 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:30:20.846901 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d7d1860_d847_410f_8e0e_3e5065167ed2.slice/crio-354ce96afe2aece4cf77d70eff1090fd9c6b0be683ca20e948cf35b5d76ec061 WatchSource:0}: Error finding container 354ce96afe2aece4cf77d70eff1090fd9c6b0be683ca20e948cf35b5d76ec061: Status 404 returned error can't find the container with id 354ce96afe2aece4cf77d70eff1090fd9c6b0be683ca20e948cf35b5d76ec061 Nov 02 17:30:21.658320 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:21.658279 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f987684bb-8llbv" event={"ID":"0d7d1860-d847-410f-8e0e-3e5065167ed2","Type":"ContainerStarted","Data":"685f5bfad7603b581a4cce6ccdf951cb0de1202bb811d50fdb381d689d14af45"} Nov 02 17:30:21.658708 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:21.658342 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f987684bb-8llbv" event={"ID":"0d7d1860-d847-410f-8e0e-3e5065167ed2","Type":"ContainerStarted","Data":"354ce96afe2aece4cf77d70eff1090fd9c6b0be683ca20e948cf35b5d76ec061"} Nov 02 17:30:21.717054 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:21.717004 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7f987684bb-8llbv" podStartSLOduration=1.716987788 podStartE2EDuration="1.716987788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:30:21.704590642 +0000 UTC m=+39.399946135" watchObservedRunningTime="2026-11-02 17:30:21.716987788 +0000 UTC m=+39.412343261" Nov 02 17:30:27.639517 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:27.639427 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f987684bb-8llbv"] Nov 02 17:30:27.680155 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:27.680122 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-58f656c7fb-bzfsh"] Nov 02 17:30:27.682375 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:27.682347 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58f656c7fb-bzfsh" Nov 02 17:30:27.708528 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:27.708499 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58f656c7fb-bzfsh"] Nov 02 17:30:27.796225 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:27.796189 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aaa8718c-561e-4164-aec5-ac3bba86706b-oauth-serving-cert\") pod \"console-58f656c7fb-bzfsh\" (UID: \"aaa8718c-561e-4164-aec5-ac3bba86706b\") " pod="openshift-console/console-58f656c7fb-bzfsh" Nov 02 17:30:27.796225 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:27.796231 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g26n2\" (UniqueName: \"kubernetes.io/projected/aaa8718c-561e-4164-aec5-ac3bba86706b-kube-api-access-g26n2\") pod \"console-58f656c7fb-bzfsh\" (UID: \"aaa8718c-561e-4164-aec5-ac3bba86706b\") " pod="openshift-console/console-58f656c7fb-bzfsh" Nov 02 17:30:27.796479 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:27.796250 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aaa8718c-561e-4164-aec5-ac3bba86706b-console-config\") pod \"console-58f656c7fb-bzfsh\" (UID: \"aaa8718c-561e-4164-aec5-ac3bba86706b\") " pod="openshift-console/console-58f656c7fb-bzfsh" Nov 02 17:30:27.796479 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:27.796401 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aaa8718c-561e-4164-aec5-ac3bba86706b-tmp\") pod \"console-58f656c7fb-bzfsh\" (UID: \"aaa8718c-561e-4164-aec5-ac3bba86706b\") " pod="openshift-console/console-58f656c7fb-bzfsh" Nov 02 17:30:27.796479 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:27.796434 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa8718c-561e-4164-aec5-ac3bba86706b-trusted-ca-bundle\") pod \"console-58f656c7fb-bzfsh\" (UID: \"aaa8718c-561e-4164-aec5-ac3bba86706b\") " pod="openshift-console/console-58f656c7fb-bzfsh" Nov 02 17:30:27.796479 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:27.796464 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aaa8718c-561e-4164-aec5-ac3bba86706b-console-serving-cert\") pod \"console-58f656c7fb-bzfsh\" (UID: \"aaa8718c-561e-4164-aec5-ac3bba86706b\") " pod="openshift-console/console-58f656c7fb-bzfsh" Nov 02 17:30:27.796625 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:27.796487 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa8718c-561e-4164-aec5-ac3bba86706b-service-ca\") pod \"console-58f656c7fb-bzfsh\" (UID: \"aaa8718c-561e-4164-aec5-ac3bba86706b\") " pod="openshift-console/console-58f656c7fb-bzfsh" Nov 02 17:30:27.796625 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:27.796566 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aaa8718c-561e-4164-aec5-ac3bba86706b-console-oauth-config\") pod \"console-58f656c7fb-bzfsh\" (UID: \"aaa8718c-561e-4164-aec5-ac3bba86706b\") " pod="openshift-console/console-58f656c7fb-bzfsh" Nov 02 17:30:27.897481 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:27.897395 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g26n2\" (UniqueName: \"kubernetes.io/projected/aaa8718c-561e-4164-aec5-ac3bba86706b-kube-api-access-g26n2\") pod \"console-58f656c7fb-bzfsh\" (UID: \"aaa8718c-561e-4164-aec5-ac3bba86706b\") " pod="openshift-console/console-58f656c7fb-bzfsh" Nov 02 17:30:27.897481 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:27.897440 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aaa8718c-561e-4164-aec5-ac3bba86706b-console-config\") pod \"console-58f656c7fb-bzfsh\" (UID: \"aaa8718c-561e-4164-aec5-ac3bba86706b\") " pod="openshift-console/console-58f656c7fb-bzfsh" Nov 02 17:30:27.897691 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:27.897485 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aaa8718c-561e-4164-aec5-ac3bba86706b-tmp\") pod \"console-58f656c7fb-bzfsh\" (UID: \"aaa8718c-561e-4164-aec5-ac3bba86706b\") " pod="openshift-console/console-58f656c7fb-bzfsh" Nov 02 17:30:27.897691 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:27.897507 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa8718c-561e-4164-aec5-ac3bba86706b-trusted-ca-bundle\") pod \"console-58f656c7fb-bzfsh\" (UID: \"aaa8718c-561e-4164-aec5-ac3bba86706b\") " pod="openshift-console/console-58f656c7fb-bzfsh" Nov 02 17:30:27.897691 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:27.897547 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aaa8718c-561e-4164-aec5-ac3bba86706b-console-serving-cert\") pod \"console-58f656c7fb-bzfsh\" (UID: \"aaa8718c-561e-4164-aec5-ac3bba86706b\") " pod="openshift-console/console-58f656c7fb-bzfsh" Nov 02 17:30:27.897691 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:27.897573 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa8718c-561e-4164-aec5-ac3bba86706b-service-ca\") pod \"console-58f656c7fb-bzfsh\" (UID: \"aaa8718c-561e-4164-aec5-ac3bba86706b\") " pod="openshift-console/console-58f656c7fb-bzfsh" Nov 02 17:30:27.897890 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:27.897714 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aaa8718c-561e-4164-aec5-ac3bba86706b-console-oauth-config\") pod \"console-58f656c7fb-bzfsh\" (UID: \"aaa8718c-561e-4164-aec5-ac3bba86706b\") " pod="openshift-console/console-58f656c7fb-bzfsh" Nov 02 17:30:27.897890 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:27.897798 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aaa8718c-561e-4164-aec5-ac3bba86706b-oauth-serving-cert\") pod \"console-58f656c7fb-bzfsh\" (UID: \"aaa8718c-561e-4164-aec5-ac3bba86706b\") " pod="openshift-console/console-58f656c7fb-bzfsh" Nov 02 17:30:27.897970 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:27.897953 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aaa8718c-561e-4164-aec5-ac3bba86706b-tmp\") pod \"console-58f656c7fb-bzfsh\" (UID: \"aaa8718c-561e-4164-aec5-ac3bba86706b\") " pod="openshift-console/console-58f656c7fb-bzfsh" Nov 02 17:30:27.898457 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:27.898436 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa8718c-561e-4164-aec5-ac3bba86706b-service-ca\") pod \"console-58f656c7fb-bzfsh\" (UID: \"aaa8718c-561e-4164-aec5-ac3bba86706b\") " pod="openshift-console/console-58f656c7fb-bzfsh" Nov 02 17:30:27.898576 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:27.898466 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aaa8718c-561e-4164-aec5-ac3bba86706b-console-config\") pod \"console-58f656c7fb-bzfsh\" (UID: \"aaa8718c-561e-4164-aec5-ac3bba86706b\") " pod="openshift-console/console-58f656c7fb-bzfsh" Nov 02 17:30:27.898619 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:27.898570 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa8718c-561e-4164-aec5-ac3bba86706b-trusted-ca-bundle\") pod \"console-58f656c7fb-bzfsh\" (UID: \"aaa8718c-561e-4164-aec5-ac3bba86706b\") " pod="openshift-console/console-58f656c7fb-bzfsh" Nov 02 17:30:27.898619 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:27.898570 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aaa8718c-561e-4164-aec5-ac3bba86706b-oauth-serving-cert\") pod \"console-58f656c7fb-bzfsh\" (UID: \"aaa8718c-561e-4164-aec5-ac3bba86706b\") " pod="openshift-console/console-58f656c7fb-bzfsh" Nov 02 17:30:27.900044 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:27.900016 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aaa8718c-561e-4164-aec5-ac3bba86706b-console-serving-cert\") pod \"console-58f656c7fb-bzfsh\" (UID: \"aaa8718c-561e-4164-aec5-ac3bba86706b\") " pod="openshift-console/console-58f656c7fb-bzfsh" Nov 02 17:30:27.900171 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:27.900155 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aaa8718c-561e-4164-aec5-ac3bba86706b-console-oauth-config\") pod \"console-58f656c7fb-bzfsh\" (UID: \"aaa8718c-561e-4164-aec5-ac3bba86706b\") " pod="openshift-console/console-58f656c7fb-bzfsh" Nov 02 17:30:27.921768 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:27.921746 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g26n2\" (UniqueName: \"kubernetes.io/projected/aaa8718c-561e-4164-aec5-ac3bba86706b-kube-api-access-g26n2\") pod \"console-58f656c7fb-bzfsh\" (UID: \"aaa8718c-561e-4164-aec5-ac3bba86706b\") " pod="openshift-console/console-58f656c7fb-bzfsh" Nov 02 17:30:27.993835 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:27.993799 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58f656c7fb-bzfsh" Nov 02 17:30:28.305544 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:28.305518 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58f656c7fb-bzfsh"] Nov 02 17:30:28.306092 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:30:28.306059 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa8718c_561e_4164_aec5_ac3bba86706b.slice/crio-193d5a5151e3e9ee6f4c5c848b8f95ab7b1cb4f38d5e28a7fc0bfa89086648f3 WatchSource:0}: Error finding container 193d5a5151e3e9ee6f4c5c848b8f95ab7b1cb4f38d5e28a7fc0bfa89086648f3: Status 404 returned error can't find the container with id 193d5a5151e3e9ee6f4c5c848b8f95ab7b1cb4f38d5e28a7fc0bfa89086648f3 Nov 02 17:30:28.686624 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:28.686547 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58f656c7fb-bzfsh" event={"ID":"aaa8718c-561e-4164-aec5-ac3bba86706b","Type":"ContainerStarted","Data":"63259ff3a373fae7fcba2e48fecefc140f56d0be535a7b92b052067f00ac3010"} Nov 02 17:30:28.686624 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:28.686603 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58f656c7fb-bzfsh" event={"ID":"aaa8718c-561e-4164-aec5-ac3bba86706b","Type":"ContainerStarted","Data":"193d5a5151e3e9ee6f4c5c848b8f95ab7b1cb4f38d5e28a7fc0bfa89086648f3"} Nov 02 17:30:28.732004 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:28.731954 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-58f656c7fb-bzfsh" podStartSLOduration=1.731937695 podStartE2EDuration="1.731937695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:30:28.730756778 +0000 UTC m=+46.426112271" watchObservedRunningTime="2026-11-02 17:30:28.731937695 +0000 UTC m=+46.427293167" Nov 02 17:30:30.552873 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:30.552832 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f987684bb-8llbv" Nov 02 17:30:37.993913 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:37.993877 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58f656c7fb-bzfsh" Nov 02 17:30:37.994432 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:37.993972 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58f656c7fb-bzfsh" Nov 02 17:30:37.998988 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:37.998952 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58f656c7fb-bzfsh" Nov 02 17:30:38.723494 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:38.723462 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58f656c7fb-bzfsh" Nov 02 17:30:38.805586 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:38.805547 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b8c96d88b-cnbq8"] Nov 02 17:30:39.181251 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:30:39.181207 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:30:52.658959 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:52.658907 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7f987684bb-8llbv" podUID="0d7d1860-d847-410f-8e0e-3e5065167ed2" containerName="console" containerID="cri-o://685f5bfad7603b581a4cce6ccdf951cb0de1202bb811d50fdb381d689d14af45" gracePeriod=15 Nov 02 17:30:52.893599 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:52.893577 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f987684bb-8llbv" Nov 02 17:30:52.996562 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:52.996492 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d7d1860-d847-410f-8e0e-3e5065167ed2-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d7d1860-d847-410f-8e0e-3e5065167ed2-tmp\") pod \"0d7d1860-d847-410f-8e0e-3e5065167ed2\" (UID: \"0d7d1860-d847-410f-8e0e-3e5065167ed2\") " Nov 02 17:30:52.996562 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:52.996527 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d7d1860-d847-410f-8e0e-3e5065167ed2-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d7d1860-d847-410f-8e0e-3e5065167ed2-console-oauth-config\") pod \"0d7d1860-d847-410f-8e0e-3e5065167ed2\" (UID: \"0d7d1860-d847-410f-8e0e-3e5065167ed2\") " Nov 02 17:30:52.996562 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:52.996555 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d7d1860-d847-410f-8e0e-3e5065167ed2-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d7d1860-d847-410f-8e0e-3e5065167ed2-console-serving-cert\") pod \"0d7d1860-d847-410f-8e0e-3e5065167ed2\" (UID: \"0d7d1860-d847-410f-8e0e-3e5065167ed2\") " Nov 02 17:30:52.996833 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:52.996575 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d7d1860-d847-410f-8e0e-3e5065167ed2-kube-api-access-hzqqb\" (UniqueName: \"kubernetes.io/projected/0d7d1860-d847-410f-8e0e-3e5065167ed2-kube-api-access-hzqqb\") pod \"0d7d1860-d847-410f-8e0e-3e5065167ed2\" (UID: \"0d7d1860-d847-410f-8e0e-3e5065167ed2\") " Nov 02 17:30:52.996833 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:52.996594 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d7d1860-d847-410f-8e0e-3e5065167ed2-console-config\" (UniqueName: \"kubernetes.io/configmap/0d7d1860-d847-410f-8e0e-3e5065167ed2-console-config\") pod \"0d7d1860-d847-410f-8e0e-3e5065167ed2\" (UID: \"0d7d1860-d847-410f-8e0e-3e5065167ed2\") " Nov 02 17:30:52.996833 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:52.996637 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d7d1860-d847-410f-8e0e-3e5065167ed2-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d7d1860-d847-410f-8e0e-3e5065167ed2-service-ca\") pod \"0d7d1860-d847-410f-8e0e-3e5065167ed2\" (UID: \"0d7d1860-d847-410f-8e0e-3e5065167ed2\") " Nov 02 17:30:52.996833 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:52.996657 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d7d1860-d847-410f-8e0e-3e5065167ed2-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d7d1860-d847-410f-8e0e-3e5065167ed2-trusted-ca-bundle\") pod \"0d7d1860-d847-410f-8e0e-3e5065167ed2\" (UID: \"0d7d1860-d847-410f-8e0e-3e5065167ed2\") " Nov 02 17:30:52.996833 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:52.996720 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d7d1860-d847-410f-8e0e-3e5065167ed2-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d7d1860-d847-410f-8e0e-3e5065167ed2-oauth-serving-cert\") pod \"0d7d1860-d847-410f-8e0e-3e5065167ed2\" (UID: \"0d7d1860-d847-410f-8e0e-3e5065167ed2\") " Nov 02 17:30:52.997095 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:52.996843 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d7d1860-d847-410f-8e0e-3e5065167ed2-tmp" pod "0d7d1860-d847-410f-8e0e-3e5065167ed2" (UID: "0d7d1860-d847-410f-8e0e-3e5065167ed2"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:30:52.997095 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:52.997047 4471 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d7d1860-d847-410f-8e0e-3e5065167ed2-tmp\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:30:52.997095 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:52.997054 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d7d1860-d847-410f-8e0e-3e5065167ed2-console-config" pod "0d7d1860-d847-410f-8e0e-3e5065167ed2" (UID: "0d7d1860-d847-410f-8e0e-3e5065167ed2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:30:52.997095 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:52.997068 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d7d1860-d847-410f-8e0e-3e5065167ed2-service-ca" pod "0d7d1860-d847-410f-8e0e-3e5065167ed2" (UID: "0d7d1860-d847-410f-8e0e-3e5065167ed2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:30:52.997317 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:52.997133 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d7d1860-d847-410f-8e0e-3e5065167ed2-oauth-serving-cert" pod "0d7d1860-d847-410f-8e0e-3e5065167ed2" (UID: "0d7d1860-d847-410f-8e0e-3e5065167ed2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:30:52.997433 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:52.997408 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d7d1860-d847-410f-8e0e-3e5065167ed2-trusted-ca-bundle" pod "0d7d1860-d847-410f-8e0e-3e5065167ed2" (UID: "0d7d1860-d847-410f-8e0e-3e5065167ed2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:30:52.998834 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:52.998811 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d7d1860-d847-410f-8e0e-3e5065167ed2-console-oauth-config" pod "0d7d1860-d847-410f-8e0e-3e5065167ed2" (UID: "0d7d1860-d847-410f-8e0e-3e5065167ed2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:30:52.998937 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:52.998826 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d7d1860-d847-410f-8e0e-3e5065167ed2-console-serving-cert" pod "0d7d1860-d847-410f-8e0e-3e5065167ed2" (UID: "0d7d1860-d847-410f-8e0e-3e5065167ed2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:30:52.998937 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:52.998875 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d7d1860-d847-410f-8e0e-3e5065167ed2-kube-api-access-hzqqb" pod "0d7d1860-d847-410f-8e0e-3e5065167ed2" (UID: "0d7d1860-d847-410f-8e0e-3e5065167ed2"). InnerVolumeSpecName "kube-api-access-hzqqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:30:53.098448 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:53.098413 4471 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d7d1860-d847-410f-8e0e-3e5065167ed2-console-oauth-config\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:30:53.098448 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:53.098444 4471 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d7d1860-d847-410f-8e0e-3e5065167ed2-console-serving-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:30:53.098646 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:53.098458 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzqqb\" (UniqueName: \"kubernetes.io/projected/0d7d1860-d847-410f-8e0e-3e5065167ed2-kube-api-access-hzqqb\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:30:53.098646 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:53.098471 4471 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d7d1860-d847-410f-8e0e-3e5065167ed2-console-config\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:30:53.098646 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:53.098483 4471 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d7d1860-d847-410f-8e0e-3e5065167ed2-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:30:53.098646 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:53.098494 4471 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d7d1860-d847-410f-8e0e-3e5065167ed2-trusted-ca-bundle\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:30:53.098646 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:53.098506 4471 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d7d1860-d847-410f-8e0e-3e5065167ed2-oauth-serving-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:30:53.772056 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:53.772021 4471 generic.go:356] "Generic (PLEG): container finished" podID="0d7d1860-d847-410f-8e0e-3e5065167ed2" containerID="685f5bfad7603b581a4cce6ccdf951cb0de1202bb811d50fdb381d689d14af45" exitCode=0 Nov 02 17:30:53.772583 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:53.772079 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f987684bb-8llbv" Nov 02 17:30:53.772583 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:53.772087 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f987684bb-8llbv" event={"ID":"0d7d1860-d847-410f-8e0e-3e5065167ed2","Type":"ContainerDied","Data":"685f5bfad7603b581a4cce6ccdf951cb0de1202bb811d50fdb381d689d14af45"} Nov 02 17:30:53.772583 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:53.772125 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f987684bb-8llbv" event={"ID":"0d7d1860-d847-410f-8e0e-3e5065167ed2","Type":"ContainerDied","Data":"354ce96afe2aece4cf77d70eff1090fd9c6b0be683ca20e948cf35b5d76ec061"} Nov 02 17:30:53.772583 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:53.772145 4471 scope.go:122] "RemoveContainer" containerID="685f5bfad7603b581a4cce6ccdf951cb0de1202bb811d50fdb381d689d14af45" Nov 02 17:30:53.780797 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:53.780778 4471 scope.go:122] "RemoveContainer" containerID="685f5bfad7603b581a4cce6ccdf951cb0de1202bb811d50fdb381d689d14af45" Nov 02 17:30:53.781052 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:30:53.781029 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"685f5bfad7603b581a4cce6ccdf951cb0de1202bb811d50fdb381d689d14af45\": container with ID starting with 685f5bfad7603b581a4cce6ccdf951cb0de1202bb811d50fdb381d689d14af45 not found: ID does not exist" containerID="685f5bfad7603b581a4cce6ccdf951cb0de1202bb811d50fdb381d689d14af45" Nov 02 17:30:53.781097 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:53.781062 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"685f5bfad7603b581a4cce6ccdf951cb0de1202bb811d50fdb381d689d14af45"} err="failed to get container status \"685f5bfad7603b581a4cce6ccdf951cb0de1202bb811d50fdb381d689d14af45\": rpc error: code = NotFound desc = could not find container \"685f5bfad7603b581a4cce6ccdf951cb0de1202bb811d50fdb381d689d14af45\": container with ID starting with 685f5bfad7603b581a4cce6ccdf951cb0de1202bb811d50fdb381d689d14af45 not found: ID does not exist" Nov 02 17:30:53.828356 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:53.828334 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f987684bb-8llbv"] Nov 02 17:30:53.841296 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:53.841267 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f987684bb-8llbv"] Nov 02 17:30:54.437824 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:30:54.437793 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d7d1860-d847-410f-8e0e-3e5065167ed2" path="/var/lib/kubelet/pods/0d7d1860-d847-410f-8e0e-3e5065167ed2/volumes" Nov 02 17:31:02.318431 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:02.318365 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:31:02.318933 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:02.318705 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d3111c93-5a21-4a5b-8ada-e78c73ebf841" containerName="alertmanager" containerID="cri-o://fbeb0e97cdc43442fb546ef16d8377f735480a8c3268b9a926718c1c59fc2a50" gracePeriod=120 Nov 02 17:31:02.318933 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:02.318754 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d3111c93-5a21-4a5b-8ada-e78c73ebf841" containerName="prom-label-proxy" containerID="cri-o://8fde5fc45f9ee160bff43facaeb8938828214f7b8ef8349626297ff8699cc58b" gracePeriod=120 Nov 02 17:31:02.318933 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:02.318801 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d3111c93-5a21-4a5b-8ada-e78c73ebf841" containerName="kube-rbac-proxy-metric" containerID="cri-o://e2cfb5c8e6ddf84e602e48c968759c1c22ccc6b4d68763d85a6cc13e5446222a" gracePeriod=120 Nov 02 17:31:02.319112 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:02.319027 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d3111c93-5a21-4a5b-8ada-e78c73ebf841" containerName="kube-rbac-proxy" containerID="cri-o://9076d775b7b220e47194f834f12a2320e0a8c22f295fd89ae3d3ede3b0cea5a3" gracePeriod=120 Nov 02 17:31:02.319697 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:02.319169 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d3111c93-5a21-4a5b-8ada-e78c73ebf841" containerName="config-reloader" containerID="cri-o://cd8c35bc4569aaee407d2177089b46ae0ecc260ec82d4a3f4797e7c23d05ee23" gracePeriod=120 Nov 02 17:31:02.319697 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:02.319291 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d3111c93-5a21-4a5b-8ada-e78c73ebf841" containerName="kube-rbac-proxy-web" containerID="cri-o://dc5fc6b01424704ed71b782ced304597b143c6580d98deb33cf9f62ab759a1a9" gracePeriod=120 Nov 02 17:31:02.806127 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:02.806095 4471 generic.go:356] "Generic (PLEG): container finished" podID="d3111c93-5a21-4a5b-8ada-e78c73ebf841" containerID="8fde5fc45f9ee160bff43facaeb8938828214f7b8ef8349626297ff8699cc58b" exitCode=0 Nov 02 17:31:02.806127 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:02.806121 4471 generic.go:356] "Generic (PLEG): container finished" podID="d3111c93-5a21-4a5b-8ada-e78c73ebf841" containerID="e2cfb5c8e6ddf84e602e48c968759c1c22ccc6b4d68763d85a6cc13e5446222a" exitCode=0 Nov 02 17:31:02.806127 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:02.806127 4471 generic.go:356] "Generic (PLEG): container finished" podID="d3111c93-5a21-4a5b-8ada-e78c73ebf841" containerID="9076d775b7b220e47194f834f12a2320e0a8c22f295fd89ae3d3ede3b0cea5a3" exitCode=0 Nov 02 17:31:02.806127 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:02.806133 4471 generic.go:356] "Generic (PLEG): container finished" podID="d3111c93-5a21-4a5b-8ada-e78c73ebf841" containerID="cd8c35bc4569aaee407d2177089b46ae0ecc260ec82d4a3f4797e7c23d05ee23" exitCode=0 Nov 02 17:31:02.806414 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:02.806139 4471 generic.go:356] "Generic (PLEG): container finished" podID="d3111c93-5a21-4a5b-8ada-e78c73ebf841" containerID="fbeb0e97cdc43442fb546ef16d8377f735480a8c3268b9a926718c1c59fc2a50" exitCode=0 Nov 02 17:31:02.806414 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:02.806167 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3111c93-5a21-4a5b-8ada-e78c73ebf841","Type":"ContainerDied","Data":"8fde5fc45f9ee160bff43facaeb8938828214f7b8ef8349626297ff8699cc58b"} Nov 02 17:31:02.806414 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:02.806227 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3111c93-5a21-4a5b-8ada-e78c73ebf841","Type":"ContainerDied","Data":"e2cfb5c8e6ddf84e602e48c968759c1c22ccc6b4d68763d85a6cc13e5446222a"} Nov 02 17:31:02.806414 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:02.806253 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3111c93-5a21-4a5b-8ada-e78c73ebf841","Type":"ContainerDied","Data":"9076d775b7b220e47194f834f12a2320e0a8c22f295fd89ae3d3ede3b0cea5a3"} Nov 02 17:31:02.806414 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:02.806277 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3111c93-5a21-4a5b-8ada-e78c73ebf841","Type":"ContainerDied","Data":"cd8c35bc4569aaee407d2177089b46ae0ecc260ec82d4a3f4797e7c23d05ee23"} Nov 02 17:31:02.806414 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:02.806300 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3111c93-5a21-4a5b-8ada-e78c73ebf841","Type":"ContainerDied","Data":"fbeb0e97cdc43442fb546ef16d8377f735480a8c3268b9a926718c1c59fc2a50"} Nov 02 17:31:03.559340 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.559296 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:03.679911 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.679833 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3111c93-5a21-4a5b-8ada-e78c73ebf841-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3111c93-5a21-4a5b-8ada-e78c73ebf841-metrics-client-ca\") pod \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " Nov 02 17:31:03.679911 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.679879 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3111c93-5a21-4a5b-8ada-e78c73ebf841-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3111c93-5a21-4a5b-8ada-e78c73ebf841-alertmanager-trusted-ca-bundle\") pod \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " Nov 02 17:31:03.680101 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.679916 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3111c93-5a21-4a5b-8ada-e78c73ebf841-tls-assets\" (UniqueName: \"kubernetes.io/projected/d3111c93-5a21-4a5b-8ada-e78c73ebf841-tls-assets\") pod \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " Nov 02 17:31:03.680101 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.680039 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-web-config\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-web-config\") pod \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " Nov 02 17:31:03.680101 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.680091 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-secret-alertmanager-kube-rbac-proxy-web\") pod \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " Nov 02 17:31:03.680233 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.680131 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-secret-alertmanager-kube-rbac-proxy\") pod \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " Nov 02 17:31:03.680233 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.680170 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-secret-alertmanager-main-tls\") pod \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " Nov 02 17:31:03.680233 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.680223 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3111c93-5a21-4a5b-8ada-e78c73ebf841-config-out\" (UniqueName: \"kubernetes.io/empty-dir/d3111c93-5a21-4a5b-8ada-e78c73ebf841-config-out\") pod \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " Nov 02 17:31:03.680425 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.680252 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-config-volume\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-config-volume\") pod \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " Nov 02 17:31:03.680425 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.680275 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3111c93-5a21-4a5b-8ada-e78c73ebf841-metrics-client-ca" pod "d3111c93-5a21-4a5b-8ada-e78c73ebf841" (UID: "d3111c93-5a21-4a5b-8ada-e78c73ebf841"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:31:03.680425 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.680288 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-secret-alertmanager-kube-rbac-proxy-metric\") pod \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " Nov 02 17:31:03.680425 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.680328 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3111c93-5a21-4a5b-8ada-e78c73ebf841-alertmanager-trusted-ca-bundle" pod "d3111c93-5a21-4a5b-8ada-e78c73ebf841" (UID: "d3111c93-5a21-4a5b-8ada-e78c73ebf841"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:31:03.680425 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.680340 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-cluster-tls-config\") pod \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " Nov 02 17:31:03.680425 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.680387 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3111c93-5a21-4a5b-8ada-e78c73ebf841-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d3111c93-5a21-4a5b-8ada-e78c73ebf841-alertmanager-main-db\") pod \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " Nov 02 17:31:03.680425 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.680417 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3111c93-5a21-4a5b-8ada-e78c73ebf841-kube-api-access-k8ptt\" (UniqueName: \"kubernetes.io/projected/d3111c93-5a21-4a5b-8ada-e78c73ebf841-kube-api-access-k8ptt\") pod \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\" (UID: \"d3111c93-5a21-4a5b-8ada-e78c73ebf841\") " Nov 02 17:31:03.680844 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.680673 4471 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3111c93-5a21-4a5b-8ada-e78c73ebf841-metrics-client-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:31:03.680844 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.680696 4471 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3111c93-5a21-4a5b-8ada-e78c73ebf841-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:31:03.682048 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.682019 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3111c93-5a21-4a5b-8ada-e78c73ebf841-alertmanager-main-db" pod "d3111c93-5a21-4a5b-8ada-e78c73ebf841" (UID: "d3111c93-5a21-4a5b-8ada-e78c73ebf841"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:31:03.683521 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.683480 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3111c93-5a21-4a5b-8ada-e78c73ebf841-kube-api-access-k8ptt" pod "d3111c93-5a21-4a5b-8ada-e78c73ebf841" (UID: "d3111c93-5a21-4a5b-8ada-e78c73ebf841"). InnerVolumeSpecName "kube-api-access-k8ptt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:31:03.683814 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.683779 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3111c93-5a21-4a5b-8ada-e78c73ebf841-config-out" pod "d3111c93-5a21-4a5b-8ada-e78c73ebf841" (UID: "d3111c93-5a21-4a5b-8ada-e78c73ebf841"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:31:03.684708 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.684623 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-secret-alertmanager-kube-rbac-proxy-web" pod "d3111c93-5a21-4a5b-8ada-e78c73ebf841" (UID: "d3111c93-5a21-4a5b-8ada-e78c73ebf841"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:31:03.684708 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.684668 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-secret-alertmanager-kube-rbac-proxy" pod "d3111c93-5a21-4a5b-8ada-e78c73ebf841" (UID: "d3111c93-5a21-4a5b-8ada-e78c73ebf841"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:31:03.684708 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.684690 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-config-volume" pod "d3111c93-5a21-4a5b-8ada-e78c73ebf841" (UID: "d3111c93-5a21-4a5b-8ada-e78c73ebf841"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:31:03.685105 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.685016 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-secret-alertmanager-kube-rbac-proxy-metric" pod "d3111c93-5a21-4a5b-8ada-e78c73ebf841" (UID: "d3111c93-5a21-4a5b-8ada-e78c73ebf841"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:31:03.685105 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.685031 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-secret-alertmanager-main-tls" pod "d3111c93-5a21-4a5b-8ada-e78c73ebf841" (UID: "d3111c93-5a21-4a5b-8ada-e78c73ebf841"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:31:03.685328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.685256 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3111c93-5a21-4a5b-8ada-e78c73ebf841-tls-assets" pod "d3111c93-5a21-4a5b-8ada-e78c73ebf841" (UID: "d3111c93-5a21-4a5b-8ada-e78c73ebf841"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:31:03.688597 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.688501 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-cluster-tls-config" pod "d3111c93-5a21-4a5b-8ada-e78c73ebf841" (UID: "d3111c93-5a21-4a5b-8ada-e78c73ebf841"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:31:03.695575 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.695545 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-web-config" pod "d3111c93-5a21-4a5b-8ada-e78c73ebf841" (UID: "d3111c93-5a21-4a5b-8ada-e78c73ebf841"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:31:03.781659 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.781625 4471 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d3111c93-5a21-4a5b-8ada-e78c73ebf841-config-out\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:31:03.781659 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.781652 4471 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:31:03.781870 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.781668 4471 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:31:03.781870 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.781684 4471 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-cluster-tls-config\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:31:03.781870 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.781697 4471 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d3111c93-5a21-4a5b-8ada-e78c73ebf841-alertmanager-main-db\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:31:03.781870 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.781709 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8ptt\" (UniqueName: \"kubernetes.io/projected/d3111c93-5a21-4a5b-8ada-e78c73ebf841-kube-api-access-k8ptt\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:31:03.781870 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.781725 4471 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d3111c93-5a21-4a5b-8ada-e78c73ebf841-tls-assets\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:31:03.781870 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.781738 4471 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-web-config\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:31:03.781870 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.781750 4471 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:31:03.781870 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.781763 4471 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:31:03.781870 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.781776 4471 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d3111c93-5a21-4a5b-8ada-e78c73ebf841-secret-alertmanager-main-tls\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:31:03.812979 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.812941 4471 generic.go:356] "Generic (PLEG): container finished" podID="d3111c93-5a21-4a5b-8ada-e78c73ebf841" containerID="dc5fc6b01424704ed71b782ced304597b143c6580d98deb33cf9f62ab759a1a9" exitCode=0 Nov 02 17:31:03.813109 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.813007 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3111c93-5a21-4a5b-8ada-e78c73ebf841","Type":"ContainerDied","Data":"dc5fc6b01424704ed71b782ced304597b143c6580d98deb33cf9f62ab759a1a9"} Nov 02 17:31:03.813109 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.813021 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:03.813109 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.813061 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3111c93-5a21-4a5b-8ada-e78c73ebf841","Type":"ContainerDied","Data":"c28a4819fe5cfd14fcc3706e868411d5794be98e628a2a0f473b7c5ee212e495"} Nov 02 17:31:03.813109 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.813092 4471 scope.go:122] "RemoveContainer" containerID="8fde5fc45f9ee160bff43facaeb8938828214f7b8ef8349626297ff8699cc58b" Nov 02 17:31:03.820455 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.820286 4471 scope.go:122] "RemoveContainer" containerID="e2cfb5c8e6ddf84e602e48c968759c1c22ccc6b4d68763d85a6cc13e5446222a" Nov 02 17:31:03.826803 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.826787 4471 scope.go:122] "RemoveContainer" containerID="9076d775b7b220e47194f834f12a2320e0a8c22f295fd89ae3d3ede3b0cea5a3" Nov 02 17:31:03.829842 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.829818 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-b8c96d88b-cnbq8" podUID="80554576-a567-4705-acd5-2b933615e02d" containerName="console" containerID="cri-o://7adf2f7ce4a21b314c5f3c817afd36573102cc8850d49796eb6ee835f3088b6c" gracePeriod=15 Nov 02 17:31:03.833728 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.833711 4471 scope.go:122] "RemoveContainer" containerID="dc5fc6b01424704ed71b782ced304597b143c6580d98deb33cf9f62ab759a1a9" Nov 02 17:31:03.839920 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.839899 4471 scope.go:122] "RemoveContainer" containerID="cd8c35bc4569aaee407d2177089b46ae0ecc260ec82d4a3f4797e7c23d05ee23" Nov 02 17:31:03.847606 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.847593 4471 scope.go:122] "RemoveContainer" containerID="fbeb0e97cdc43442fb546ef16d8377f735480a8c3268b9a926718c1c59fc2a50" Nov 02 17:31:03.869890 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.869866 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:31:03.876342 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.876299 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:31:03.911429 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.911402 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:31:03.911647 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.911636 4471 cpu_manager.go:414] "containerMap: removing container" podUID="d3111c93-5a21-4a5b-8ada-e78c73ebf841" containerName="prom-label-proxy" Nov 02 17:31:03.911686 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.911648 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3111c93-5a21-4a5b-8ada-e78c73ebf841" containerName="prom-label-proxy" Nov 02 17:31:03.911686 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.911661 4471 cpu_manager.go:414] "containerMap: removing container" podUID="d3111c93-5a21-4a5b-8ada-e78c73ebf841" containerName="kube-rbac-proxy-web" Nov 02 17:31:03.911686 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.911666 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3111c93-5a21-4a5b-8ada-e78c73ebf841" containerName="kube-rbac-proxy-web" Nov 02 17:31:03.911686 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.911675 4471 cpu_manager.go:414] "containerMap: removing container" podUID="d3111c93-5a21-4a5b-8ada-e78c73ebf841" containerName="kube-rbac-proxy-metric" Nov 02 17:31:03.911686 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.911680 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3111c93-5a21-4a5b-8ada-e78c73ebf841" containerName="kube-rbac-proxy-metric" Nov 02 17:31:03.911686 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.911687 4471 cpu_manager.go:414] "containerMap: removing container" podUID="d3111c93-5a21-4a5b-8ada-e78c73ebf841" containerName="config-reloader" Nov 02 17:31:03.911863 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.911691 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3111c93-5a21-4a5b-8ada-e78c73ebf841" containerName="config-reloader" Nov 02 17:31:03.911863 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.911701 4471 cpu_manager.go:414] "containerMap: removing container" podUID="d3111c93-5a21-4a5b-8ada-e78c73ebf841" containerName="kube-rbac-proxy" Nov 02 17:31:03.911863 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.911707 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3111c93-5a21-4a5b-8ada-e78c73ebf841" containerName="kube-rbac-proxy" Nov 02 17:31:03.911863 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.911716 4471 cpu_manager.go:414] "containerMap: removing container" podUID="d3111c93-5a21-4a5b-8ada-e78c73ebf841" containerName="init-config-reloader" Nov 02 17:31:03.911863 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.911721 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3111c93-5a21-4a5b-8ada-e78c73ebf841" containerName="init-config-reloader" Nov 02 17:31:03.911863 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.911728 4471 cpu_manager.go:414] "containerMap: removing container" podUID="0d7d1860-d847-410f-8e0e-3e5065167ed2" containerName="console" Nov 02 17:31:03.911863 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.911732 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d7d1860-d847-410f-8e0e-3e5065167ed2" containerName="console" Nov 02 17:31:03.911863 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.911740 4471 cpu_manager.go:414] "containerMap: removing container" podUID="d3111c93-5a21-4a5b-8ada-e78c73ebf841" containerName="alertmanager" Nov 02 17:31:03.911863 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.911744 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3111c93-5a21-4a5b-8ada-e78c73ebf841" containerName="alertmanager" Nov 02 17:31:03.911863 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.911791 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3111c93-5a21-4a5b-8ada-e78c73ebf841" containerName="kube-rbac-proxy" Nov 02 17:31:03.911863 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.911799 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3111c93-5a21-4a5b-8ada-e78c73ebf841" containerName="prom-label-proxy" Nov 02 17:31:03.911863 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.911805 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d7d1860-d847-410f-8e0e-3e5065167ed2" containerName="console" Nov 02 17:31:03.911863 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.911811 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3111c93-5a21-4a5b-8ada-e78c73ebf841" containerName="kube-rbac-proxy-web" Nov 02 17:31:03.911863 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.911817 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3111c93-5a21-4a5b-8ada-e78c73ebf841" containerName="kube-rbac-proxy-metric" Nov 02 17:31:03.911863 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.911823 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3111c93-5a21-4a5b-8ada-e78c73ebf841" containerName="alertmanager" Nov 02 17:31:03.911863 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.911828 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3111c93-5a21-4a5b-8ada-e78c73ebf841" containerName="config-reloader" Nov 02 17:31:03.916355 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.916329 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:03.938051 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.934506 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 17:31:03.938051 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.934942 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 17:31:03.938051 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.935021 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-h948w\"" Nov 02 17:31:03.938249 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.938081 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 17:31:03.943417 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.943398 4471 scope.go:122] "RemoveContainer" containerID="ec98c3a7e7a0200db86f3bfa67292ee33789fef2ac095c779739a53d2ee5f04c" Nov 02 17:31:03.945360 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.945300 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 17:31:03.945484 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.945465 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 17:31:03.945600 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.945526 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 17:31:03.946217 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.946194 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 17:31:03.946217 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.946196 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 17:31:03.952230 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.952201 4471 scope.go:122] "RemoveContainer" containerID="8fde5fc45f9ee160bff43facaeb8938828214f7b8ef8349626297ff8699cc58b" Nov 02 17:31:03.952538 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:31:03.952507 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fde5fc45f9ee160bff43facaeb8938828214f7b8ef8349626297ff8699cc58b\": container with ID starting with 8fde5fc45f9ee160bff43facaeb8938828214f7b8ef8349626297ff8699cc58b not found: ID does not exist" containerID="8fde5fc45f9ee160bff43facaeb8938828214f7b8ef8349626297ff8699cc58b" Nov 02 17:31:03.952621 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.952543 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fde5fc45f9ee160bff43facaeb8938828214f7b8ef8349626297ff8699cc58b"} err="failed to get container status \"8fde5fc45f9ee160bff43facaeb8938828214f7b8ef8349626297ff8699cc58b\": rpc error: code = NotFound desc = could not find container \"8fde5fc45f9ee160bff43facaeb8938828214f7b8ef8349626297ff8699cc58b\": container with ID starting with 8fde5fc45f9ee160bff43facaeb8938828214f7b8ef8349626297ff8699cc58b not found: ID does not exist" Nov 02 17:31:03.952621 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.952570 4471 scope.go:122] "RemoveContainer" containerID="e2cfb5c8e6ddf84e602e48c968759c1c22ccc6b4d68763d85a6cc13e5446222a" Nov 02 17:31:03.952855 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:31:03.952833 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2cfb5c8e6ddf84e602e48c968759c1c22ccc6b4d68763d85a6cc13e5446222a\": container with ID starting with e2cfb5c8e6ddf84e602e48c968759c1c22ccc6b4d68763d85a6cc13e5446222a not found: ID does not exist" containerID="e2cfb5c8e6ddf84e602e48c968759c1c22ccc6b4d68763d85a6cc13e5446222a" Nov 02 17:31:03.952924 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.952864 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2cfb5c8e6ddf84e602e48c968759c1c22ccc6b4d68763d85a6cc13e5446222a"} err="failed to get container status \"e2cfb5c8e6ddf84e602e48c968759c1c22ccc6b4d68763d85a6cc13e5446222a\": rpc error: code = NotFound desc = could not find container \"e2cfb5c8e6ddf84e602e48c968759c1c22ccc6b4d68763d85a6cc13e5446222a\": container with ID starting with e2cfb5c8e6ddf84e602e48c968759c1c22ccc6b4d68763d85a6cc13e5446222a not found: ID does not exist" Nov 02 17:31:03.952924 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.952886 4471 scope.go:122] "RemoveContainer" containerID="9076d775b7b220e47194f834f12a2320e0a8c22f295fd89ae3d3ede3b0cea5a3" Nov 02 17:31:03.953118 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:31:03.953099 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9076d775b7b220e47194f834f12a2320e0a8c22f295fd89ae3d3ede3b0cea5a3\": container with ID starting with 9076d775b7b220e47194f834f12a2320e0a8c22f295fd89ae3d3ede3b0cea5a3 not found: ID does not exist" containerID="9076d775b7b220e47194f834f12a2320e0a8c22f295fd89ae3d3ede3b0cea5a3" Nov 02 17:31:03.953184 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.953123 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9076d775b7b220e47194f834f12a2320e0a8c22f295fd89ae3d3ede3b0cea5a3"} err="failed to get container status \"9076d775b7b220e47194f834f12a2320e0a8c22f295fd89ae3d3ede3b0cea5a3\": rpc error: code = NotFound desc = could not find container \"9076d775b7b220e47194f834f12a2320e0a8c22f295fd89ae3d3ede3b0cea5a3\": container with ID starting with 9076d775b7b220e47194f834f12a2320e0a8c22f295fd89ae3d3ede3b0cea5a3 not found: ID does not exist" Nov 02 17:31:03.953184 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.953144 4471 scope.go:122] "RemoveContainer" containerID="dc5fc6b01424704ed71b782ced304597b143c6580d98deb33cf9f62ab759a1a9" Nov 02 17:31:03.953385 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:31:03.953364 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc5fc6b01424704ed71b782ced304597b143c6580d98deb33cf9f62ab759a1a9\": container with ID starting with dc5fc6b01424704ed71b782ced304597b143c6580d98deb33cf9f62ab759a1a9 not found: ID does not exist" containerID="dc5fc6b01424704ed71b782ced304597b143c6580d98deb33cf9f62ab759a1a9" Nov 02 17:31:03.953447 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.953390 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc5fc6b01424704ed71b782ced304597b143c6580d98deb33cf9f62ab759a1a9"} err="failed to get container status \"dc5fc6b01424704ed71b782ced304597b143c6580d98deb33cf9f62ab759a1a9\": rpc error: code = NotFound desc = could not find container \"dc5fc6b01424704ed71b782ced304597b143c6580d98deb33cf9f62ab759a1a9\": container with ID starting with dc5fc6b01424704ed71b782ced304597b143c6580d98deb33cf9f62ab759a1a9 not found: ID does not exist" Nov 02 17:31:03.953447 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.953407 4471 scope.go:122] "RemoveContainer" containerID="cd8c35bc4569aaee407d2177089b46ae0ecc260ec82d4a3f4797e7c23d05ee23" Nov 02 17:31:03.953642 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:31:03.953626 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd8c35bc4569aaee407d2177089b46ae0ecc260ec82d4a3f4797e7c23d05ee23\": container with ID starting with cd8c35bc4569aaee407d2177089b46ae0ecc260ec82d4a3f4797e7c23d05ee23 not found: ID does not exist" containerID="cd8c35bc4569aaee407d2177089b46ae0ecc260ec82d4a3f4797e7c23d05ee23" Nov 02 17:31:03.953708 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.953643 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd8c35bc4569aaee407d2177089b46ae0ecc260ec82d4a3f4797e7c23d05ee23"} err="failed to get container status \"cd8c35bc4569aaee407d2177089b46ae0ecc260ec82d4a3f4797e7c23d05ee23\": rpc error: code = NotFound desc = could not find container \"cd8c35bc4569aaee407d2177089b46ae0ecc260ec82d4a3f4797e7c23d05ee23\": container with ID starting with cd8c35bc4569aaee407d2177089b46ae0ecc260ec82d4a3f4797e7c23d05ee23 not found: ID does not exist" Nov 02 17:31:03.953708 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.953654 4471 scope.go:122] "RemoveContainer" containerID="fbeb0e97cdc43442fb546ef16d8377f735480a8c3268b9a926718c1c59fc2a50" Nov 02 17:31:03.953854 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:31:03.953836 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbeb0e97cdc43442fb546ef16d8377f735480a8c3268b9a926718c1c59fc2a50\": container with ID starting with fbeb0e97cdc43442fb546ef16d8377f735480a8c3268b9a926718c1c59fc2a50 not found: ID does not exist" containerID="fbeb0e97cdc43442fb546ef16d8377f735480a8c3268b9a926718c1c59fc2a50" Nov 02 17:31:03.953915 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.953859 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbeb0e97cdc43442fb546ef16d8377f735480a8c3268b9a926718c1c59fc2a50"} err="failed to get container status \"fbeb0e97cdc43442fb546ef16d8377f735480a8c3268b9a926718c1c59fc2a50\": rpc error: code = NotFound desc = could not find container \"fbeb0e97cdc43442fb546ef16d8377f735480a8c3268b9a926718c1c59fc2a50\": container with ID starting with fbeb0e97cdc43442fb546ef16d8377f735480a8c3268b9a926718c1c59fc2a50 not found: ID does not exist" Nov 02 17:31:03.953915 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.953881 4471 scope.go:122] "RemoveContainer" containerID="ec98c3a7e7a0200db86f3bfa67292ee33789fef2ac095c779739a53d2ee5f04c" Nov 02 17:31:03.954084 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:31:03.954069 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec98c3a7e7a0200db86f3bfa67292ee33789fef2ac095c779739a53d2ee5f04c\": container with ID starting with ec98c3a7e7a0200db86f3bfa67292ee33789fef2ac095c779739a53d2ee5f04c not found: ID does not exist" containerID="ec98c3a7e7a0200db86f3bfa67292ee33789fef2ac095c779739a53d2ee5f04c" Nov 02 17:31:03.954141 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.954085 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec98c3a7e7a0200db86f3bfa67292ee33789fef2ac095c779739a53d2ee5f04c"} err="failed to get container status \"ec98c3a7e7a0200db86f3bfa67292ee33789fef2ac095c779739a53d2ee5f04c\": rpc error: code = NotFound desc = could not find container \"ec98c3a7e7a0200db86f3bfa67292ee33789fef2ac095c779739a53d2ee5f04c\": container with ID starting with ec98c3a7e7a0200db86f3bfa67292ee33789fef2ac095c779739a53d2ee5f04c not found: ID does not exist" Nov 02 17:31:03.958417 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.958397 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 17:31:03.960519 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:03.960498 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:31:04.061479 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.061452 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b8c96d88b-cnbq8" Nov 02 17:31:04.084513 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.084479 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-config-volume\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.084513 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.084514 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-web-config\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.084706 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.084533 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lfxn\" (UniqueName: \"kubernetes.io/projected/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-kube-api-access-6lfxn\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.084706 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.084609 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.084706 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.084694 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.084844 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.084717 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.084844 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.084799 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.084844 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.084831 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.084944 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.084850 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.084944 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.084930 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.085015 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.084953 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-config-out\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.085015 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.084971 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.085015 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.084988 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-tls-assets\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.185337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.185288 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80554576-a567-4705-acd5-2b933615e02d-console-config\" (UniqueName: \"kubernetes.io/configmap/80554576-a567-4705-acd5-2b933615e02d-console-config\") pod \"80554576-a567-4705-acd5-2b933615e02d\" (UID: \"80554576-a567-4705-acd5-2b933615e02d\") " Nov 02 17:31:04.185488 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.185363 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80554576-a567-4705-acd5-2b933615e02d-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80554576-a567-4705-acd5-2b933615e02d-oauth-serving-cert\") pod \"80554576-a567-4705-acd5-2b933615e02d\" (UID: \"80554576-a567-4705-acd5-2b933615e02d\") " Nov 02 17:31:04.185488 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.185386 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80554576-a567-4705-acd5-2b933615e02d-kube-api-access-qt777\" (UniqueName: \"kubernetes.io/projected/80554576-a567-4705-acd5-2b933615e02d-kube-api-access-qt777\") pod \"80554576-a567-4705-acd5-2b933615e02d\" (UID: \"80554576-a567-4705-acd5-2b933615e02d\") " Nov 02 17:31:04.185488 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.185421 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80554576-a567-4705-acd5-2b933615e02d-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80554576-a567-4705-acd5-2b933615e02d-console-serving-cert\") pod \"80554576-a567-4705-acd5-2b933615e02d\" (UID: \"80554576-a567-4705-acd5-2b933615e02d\") " Nov 02 17:31:04.185488 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.185447 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80554576-a567-4705-acd5-2b933615e02d-tmp\" (UniqueName: \"kubernetes.io/empty-dir/80554576-a567-4705-acd5-2b933615e02d-tmp\") pod \"80554576-a567-4705-acd5-2b933615e02d\" (UID: \"80554576-a567-4705-acd5-2b933615e02d\") " Nov 02 17:31:04.185488 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.185468 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80554576-a567-4705-acd5-2b933615e02d-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80554576-a567-4705-acd5-2b933615e02d-console-oauth-config\") pod \"80554576-a567-4705-acd5-2b933615e02d\" (UID: \"80554576-a567-4705-acd5-2b933615e02d\") " Nov 02 17:31:04.185799 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.185500 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80554576-a567-4705-acd5-2b933615e02d-service-ca\" (UniqueName: \"kubernetes.io/configmap/80554576-a567-4705-acd5-2b933615e02d-service-ca\") pod \"80554576-a567-4705-acd5-2b933615e02d\" (UID: \"80554576-a567-4705-acd5-2b933615e02d\") " Nov 02 17:31:04.185799 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.185597 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.185799 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.185628 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-config-out\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.185799 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.185657 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.185799 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.185682 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-tls-assets\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.185799 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.185727 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-config-volume\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.185799 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.185752 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-web-config\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.185799 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.185777 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lfxn\" (UniqueName: \"kubernetes.io/projected/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-kube-api-access-6lfxn\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.186182 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.185806 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.186182 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.185808 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80554576-a567-4705-acd5-2b933615e02d-tmp" pod "80554576-a567-4705-acd5-2b933615e02d" (UID: "80554576-a567-4705-acd5-2b933615e02d"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:31:04.186182 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.185768 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80554576-a567-4705-acd5-2b933615e02d-console-config" pod "80554576-a567-4705-acd5-2b933615e02d" (UID: "80554576-a567-4705-acd5-2b933615e02d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:31:04.186182 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.185859 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.186182 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.185935 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.186182 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.185986 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.186182 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.186024 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.186182 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.186050 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.186182 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.186105 4471 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80554576-a567-4705-acd5-2b933615e02d-tmp\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:31:04.186182 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.186130 4471 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80554576-a567-4705-acd5-2b933615e02d-console-config\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:31:04.186690 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.186235 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80554576-a567-4705-acd5-2b933615e02d-oauth-serving-cert" pod "80554576-a567-4705-acd5-2b933615e02d" (UID: "80554576-a567-4705-acd5-2b933615e02d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:31:04.186690 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.186454 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.186799 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.186648 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80554576-a567-4705-acd5-2b933615e02d-service-ca" pod "80554576-a567-4705-acd5-2b933615e02d" (UID: "80554576-a567-4705-acd5-2b933615e02d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:31:04.187161 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.187134 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.188738 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.188686 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-config-out\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.188897 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.188816 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.189076 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.189026 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80554576-a567-4705-acd5-2b933615e02d-kube-api-access-qt777" pod "80554576-a567-4705-acd5-2b933615e02d" (UID: "80554576-a567-4705-acd5-2b933615e02d"). InnerVolumeSpecName "kube-api-access-qt777". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:31:04.189076 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.189025 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80554576-a567-4705-acd5-2b933615e02d-console-serving-cert" pod "80554576-a567-4705-acd5-2b933615e02d" (UID: "80554576-a567-4705-acd5-2b933615e02d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:31:04.189343 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.189298 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.189680 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.189661 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.189770 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.189683 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80554576-a567-4705-acd5-2b933615e02d-console-oauth-config" pod "80554576-a567-4705-acd5-2b933615e02d" (UID: "80554576-a567-4705-acd5-2b933615e02d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:31:04.190187 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.190156 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-tls-assets\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.190385 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.190366 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.190804 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.190788 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.190949 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.190929 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-web-config\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.191287 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.191263 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-config-volume\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.191359 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.191294 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.210632 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.210614 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lfxn\" (UniqueName: \"kubernetes.io/projected/00b0092d-4e1f-4bb5-97a3-4e6e7e369d82-kube-api-access-6lfxn\") pod \"alertmanager-main-0\" (UID: \"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.243423 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.243392 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:31:04.278904 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.278872 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-684568bcb7-h8wc6"] Nov 02 17:31:04.279228 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.279211 4471 cpu_manager.go:414] "containerMap: removing container" podUID="80554576-a567-4705-acd5-2b933615e02d" containerName="console" Nov 02 17:31:04.279320 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.279230 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80554576-a567-4705-acd5-2b933615e02d" containerName="console" Nov 02 17:31:04.279379 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.279338 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="80554576-a567-4705-acd5-2b933615e02d" containerName="console" Nov 02 17:31:04.283204 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.282462 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-684568bcb7-h8wc6" Nov 02 17:31:04.287042 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.287021 4471 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80554576-a567-4705-acd5-2b933615e02d-console-oauth-config\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:31:04.287042 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.287042 4471 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80554576-a567-4705-acd5-2b933615e02d-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:31:04.287213 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.287051 4471 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80554576-a567-4705-acd5-2b933615e02d-oauth-serving-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:31:04.287213 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.287060 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt777\" (UniqueName: \"kubernetes.io/projected/80554576-a567-4705-acd5-2b933615e02d-kube-api-access-qt777\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:31:04.287213 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.287069 4471 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80554576-a567-4705-acd5-2b933615e02d-console-serving-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:31:04.312216 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.311934 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-684568bcb7-h8wc6"] Nov 02 17:31:04.387565 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.387534 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e3c97d1-c520-401d-8f11-171f00bba30d-service-ca\") pod \"console-684568bcb7-h8wc6\" (UID: \"1e3c97d1-c520-401d-8f11-171f00bba30d\") " pod="openshift-console/console-684568bcb7-h8wc6" Nov 02 17:31:04.387731 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.387570 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e3c97d1-c520-401d-8f11-171f00bba30d-trusted-ca-bundle\") pod \"console-684568bcb7-h8wc6\" (UID: \"1e3c97d1-c520-401d-8f11-171f00bba30d\") " pod="openshift-console/console-684568bcb7-h8wc6" Nov 02 17:31:04.387731 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.387588 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e3c97d1-c520-401d-8f11-171f00bba30d-console-oauth-config\") pod \"console-684568bcb7-h8wc6\" (UID: \"1e3c97d1-c520-401d-8f11-171f00bba30d\") " pod="openshift-console/console-684568bcb7-h8wc6" Nov 02 17:31:04.387731 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.387609 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e3c97d1-c520-401d-8f11-171f00bba30d-console-config\") pod \"console-684568bcb7-h8wc6\" (UID: \"1e3c97d1-c520-401d-8f11-171f00bba30d\") " pod="openshift-console/console-684568bcb7-h8wc6" Nov 02 17:31:04.387731 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.387672 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e3c97d1-c520-401d-8f11-171f00bba30d-tmp\") pod \"console-684568bcb7-h8wc6\" (UID: \"1e3c97d1-c520-401d-8f11-171f00bba30d\") " pod="openshift-console/console-684568bcb7-h8wc6" Nov 02 17:31:04.387948 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.387794 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e3c97d1-c520-401d-8f11-171f00bba30d-console-serving-cert\") pod \"console-684568bcb7-h8wc6\" (UID: \"1e3c97d1-c520-401d-8f11-171f00bba30d\") " pod="openshift-console/console-684568bcb7-h8wc6" Nov 02 17:31:04.387948 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.387885 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e3c97d1-c520-401d-8f11-171f00bba30d-oauth-serving-cert\") pod \"console-684568bcb7-h8wc6\" (UID: \"1e3c97d1-c520-401d-8f11-171f00bba30d\") " pod="openshift-console/console-684568bcb7-h8wc6" Nov 02 17:31:04.387948 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.387904 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs6pj\" (UniqueName: \"kubernetes.io/projected/1e3c97d1-c520-401d-8f11-171f00bba30d-kube-api-access-xs6pj\") pod \"console-684568bcb7-h8wc6\" (UID: \"1e3c97d1-c520-401d-8f11-171f00bba30d\") " pod="openshift-console/console-684568bcb7-h8wc6" Nov 02 17:31:04.438057 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.438028 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3111c93-5a21-4a5b-8ada-e78c73ebf841" path="/var/lib/kubelet/pods/d3111c93-5a21-4a5b-8ada-e78c73ebf841/volumes" Nov 02 17:31:04.488259 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.488224 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e3c97d1-c520-401d-8f11-171f00bba30d-console-config\") pod \"console-684568bcb7-h8wc6\" (UID: \"1e3c97d1-c520-401d-8f11-171f00bba30d\") " pod="openshift-console/console-684568bcb7-h8wc6" Nov 02 17:31:04.488259 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.488257 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e3c97d1-c520-401d-8f11-171f00bba30d-tmp\") pod \"console-684568bcb7-h8wc6\" (UID: \"1e3c97d1-c520-401d-8f11-171f00bba30d\") " pod="openshift-console/console-684568bcb7-h8wc6" Nov 02 17:31:04.488457 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.488277 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e3c97d1-c520-401d-8f11-171f00bba30d-console-serving-cert\") pod \"console-684568bcb7-h8wc6\" (UID: \"1e3c97d1-c520-401d-8f11-171f00bba30d\") " pod="openshift-console/console-684568bcb7-h8wc6" Nov 02 17:31:04.488457 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.488346 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e3c97d1-c520-401d-8f11-171f00bba30d-oauth-serving-cert\") pod \"console-684568bcb7-h8wc6\" (UID: \"1e3c97d1-c520-401d-8f11-171f00bba30d\") " pod="openshift-console/console-684568bcb7-h8wc6" Nov 02 17:31:04.488457 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.488364 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs6pj\" (UniqueName: \"kubernetes.io/projected/1e3c97d1-c520-401d-8f11-171f00bba30d-kube-api-access-xs6pj\") pod \"console-684568bcb7-h8wc6\" (UID: \"1e3c97d1-c520-401d-8f11-171f00bba30d\") " pod="openshift-console/console-684568bcb7-h8wc6" Nov 02 17:31:04.488457 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.488392 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e3c97d1-c520-401d-8f11-171f00bba30d-service-ca\") pod \"console-684568bcb7-h8wc6\" (UID: \"1e3c97d1-c520-401d-8f11-171f00bba30d\") " pod="openshift-console/console-684568bcb7-h8wc6" Nov 02 17:31:04.488457 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.488408 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e3c97d1-c520-401d-8f11-171f00bba30d-trusted-ca-bundle\") pod \"console-684568bcb7-h8wc6\" (UID: \"1e3c97d1-c520-401d-8f11-171f00bba30d\") " pod="openshift-console/console-684568bcb7-h8wc6" Nov 02 17:31:04.488457 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.488429 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e3c97d1-c520-401d-8f11-171f00bba30d-console-oauth-config\") pod \"console-684568bcb7-h8wc6\" (UID: \"1e3c97d1-c520-401d-8f11-171f00bba30d\") " pod="openshift-console/console-684568bcb7-h8wc6" Nov 02 17:31:04.488762 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.488705 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e3c97d1-c520-401d-8f11-171f00bba30d-tmp\") pod \"console-684568bcb7-h8wc6\" (UID: \"1e3c97d1-c520-401d-8f11-171f00bba30d\") " pod="openshift-console/console-684568bcb7-h8wc6" Nov 02 17:31:04.489121 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.489098 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e3c97d1-c520-401d-8f11-171f00bba30d-console-config\") pod \"console-684568bcb7-h8wc6\" (UID: \"1e3c97d1-c520-401d-8f11-171f00bba30d\") " pod="openshift-console/console-684568bcb7-h8wc6" Nov 02 17:31:04.489218 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.489127 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e3c97d1-c520-401d-8f11-171f00bba30d-service-ca\") pod \"console-684568bcb7-h8wc6\" (UID: \"1e3c97d1-c520-401d-8f11-171f00bba30d\") " pod="openshift-console/console-684568bcb7-h8wc6" Nov 02 17:31:04.489280 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.489229 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e3c97d1-c520-401d-8f11-171f00bba30d-oauth-serving-cert\") pod \"console-684568bcb7-h8wc6\" (UID: \"1e3c97d1-c520-401d-8f11-171f00bba30d\") " pod="openshift-console/console-684568bcb7-h8wc6" Nov 02 17:31:04.489374 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.489297 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e3c97d1-c520-401d-8f11-171f00bba30d-trusted-ca-bundle\") pod \"console-684568bcb7-h8wc6\" (UID: \"1e3c97d1-c520-401d-8f11-171f00bba30d\") " pod="openshift-console/console-684568bcb7-h8wc6" Nov 02 17:31:04.490843 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.490815 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e3c97d1-c520-401d-8f11-171f00bba30d-console-oauth-config\") pod \"console-684568bcb7-h8wc6\" (UID: \"1e3c97d1-c520-401d-8f11-171f00bba30d\") " pod="openshift-console/console-684568bcb7-h8wc6" Nov 02 17:31:04.490943 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.490892 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e3c97d1-c520-401d-8f11-171f00bba30d-console-serving-cert\") pod \"console-684568bcb7-h8wc6\" (UID: \"1e3c97d1-c520-401d-8f11-171f00bba30d\") " pod="openshift-console/console-684568bcb7-h8wc6" Nov 02 17:31:04.512299 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.512276 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs6pj\" (UniqueName: \"kubernetes.io/projected/1e3c97d1-c520-401d-8f11-171f00bba30d-kube-api-access-xs6pj\") pod \"console-684568bcb7-h8wc6\" (UID: \"1e3c97d1-c520-401d-8f11-171f00bba30d\") " pod="openshift-console/console-684568bcb7-h8wc6" Nov 02 17:31:04.562910 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.562887 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:31:04.563226 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:31:04.563205 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b0092d_4e1f_4bb5_97a3_4e6e7e369d82.slice/crio-9673020e7193bd03949bf73f91c4540301ad5453ef5e64565336afb1fd98312a WatchSource:0}: Error finding container 9673020e7193bd03949bf73f91c4540301ad5453ef5e64565336afb1fd98312a: Status 404 returned error can't find the container with id 9673020e7193bd03949bf73f91c4540301ad5453ef5e64565336afb1fd98312a Nov 02 17:31:04.595453 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.595426 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-684568bcb7-h8wc6" Nov 02 17:31:04.818221 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.818187 4471 generic.go:356] "Generic (PLEG): container finished" podID="80554576-a567-4705-acd5-2b933615e02d" containerID="7adf2f7ce4a21b314c5f3c817afd36573102cc8850d49796eb6ee835f3088b6c" exitCode=0 Nov 02 17:31:04.818410 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.818232 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b8c96d88b-cnbq8" Nov 02 17:31:04.818410 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.818268 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b8c96d88b-cnbq8" event={"ID":"80554576-a567-4705-acd5-2b933615e02d","Type":"ContainerDied","Data":"7adf2f7ce4a21b314c5f3c817afd36573102cc8850d49796eb6ee835f3088b6c"} Nov 02 17:31:04.818410 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.818343 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b8c96d88b-cnbq8" event={"ID":"80554576-a567-4705-acd5-2b933615e02d","Type":"ContainerDied","Data":"a9c96c06d13fe39812a1b76d3d72204dec73e4a2e0bf15cd73245667fc39758a"} Nov 02 17:31:04.818410 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.818376 4471 scope.go:122] "RemoveContainer" containerID="7adf2f7ce4a21b314c5f3c817afd36573102cc8850d49796eb6ee835f3088b6c" Nov 02 17:31:04.821056 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.821033 4471 generic.go:356] "Generic (PLEG): container finished" podID="00b0092d-4e1f-4bb5-97a3-4e6e7e369d82" containerID="b85c199b2667b05e6de71d6d0a128308321b04f179a02b01c5e5479f1f8e80c0" exitCode=0 Nov 02 17:31:04.821157 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.821094 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82","Type":"ContainerDied","Data":"b85c199b2667b05e6de71d6d0a128308321b04f179a02b01c5e5479f1f8e80c0"} Nov 02 17:31:04.821229 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.821157 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82","Type":"ContainerStarted","Data":"9673020e7193bd03949bf73f91c4540301ad5453ef5e64565336afb1fd98312a"} Nov 02 17:31:04.826467 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.826443 4471 scope.go:122] "RemoveContainer" containerID="7adf2f7ce4a21b314c5f3c817afd36573102cc8850d49796eb6ee835f3088b6c" Nov 02 17:31:04.826723 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:31:04.826706 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7adf2f7ce4a21b314c5f3c817afd36573102cc8850d49796eb6ee835f3088b6c\": container with ID starting with 7adf2f7ce4a21b314c5f3c817afd36573102cc8850d49796eb6ee835f3088b6c not found: ID does not exist" containerID="7adf2f7ce4a21b314c5f3c817afd36573102cc8850d49796eb6ee835f3088b6c" Nov 02 17:31:04.826780 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.826730 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7adf2f7ce4a21b314c5f3c817afd36573102cc8850d49796eb6ee835f3088b6c"} err="failed to get container status \"7adf2f7ce4a21b314c5f3c817afd36573102cc8850d49796eb6ee835f3088b6c\": rpc error: code = NotFound desc = could not find container \"7adf2f7ce4a21b314c5f3c817afd36573102cc8850d49796eb6ee835f3088b6c\": container with ID starting with 7adf2f7ce4a21b314c5f3c817afd36573102cc8850d49796eb6ee835f3088b6c not found: ID does not exist" Nov 02 17:31:04.871894 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.871870 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b8c96d88b-cnbq8"] Nov 02 17:31:04.875927 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.875904 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b8c96d88b-cnbq8"] Nov 02 17:31:04.901440 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:04.899320 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-684568bcb7-h8wc6"] Nov 02 17:31:04.901440 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:31:04.900460 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e3c97d1_c520_401d_8f11_171f00bba30d.slice/crio-cd64fb1b66201ce2022697ebe23e6264bbb866d061b326712b5c0d6e17e9f280 WatchSource:0}: Error finding container cd64fb1b66201ce2022697ebe23e6264bbb866d061b326712b5c0d6e17e9f280: Status 404 returned error can't find the container with id cd64fb1b66201ce2022697ebe23e6264bbb866d061b326712b5c0d6e17e9f280 Nov 02 17:31:05.826665 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:05.826626 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-684568bcb7-h8wc6" event={"ID":"1e3c97d1-c520-401d-8f11-171f00bba30d","Type":"ContainerStarted","Data":"2a4b4e72d24aececf4c66756a2e5bae9079950afe214d5a89e7ed8198d6af52d"} Nov 02 17:31:05.827111 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:05.826676 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-684568bcb7-h8wc6" event={"ID":"1e3c97d1-c520-401d-8f11-171f00bba30d","Type":"ContainerStarted","Data":"cd64fb1b66201ce2022697ebe23e6264bbb866d061b326712b5c0d6e17e9f280"} Nov 02 17:31:05.830683 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:05.830661 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82","Type":"ContainerStarted","Data":"29261f5d7eaede8afb0dae28d4ccf28052a3c14c2c85479d96776de6307d4b32"} Nov 02 17:31:05.830803 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:05.830712 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82","Type":"ContainerStarted","Data":"22b045850235bc1c874311b52624c0c7e3ef4bc4c0cec20d247989b272a708a0"} Nov 02 17:31:05.830803 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:05.830737 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82","Type":"ContainerStarted","Data":"0d647dc92d34cd592872cdb5f8639870fd8eeac3affdee8dae382e98934e21e8"} Nov 02 17:31:05.830803 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:05.830761 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82","Type":"ContainerStarted","Data":"d651117b6ce45ace0e2e1b189bbec6ba908dc6df168272f37bcb641b3a76a3ae"} Nov 02 17:31:05.830803 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:05.830790 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82","Type":"ContainerStarted","Data":"316e3137e32620a80d140c705a5b69d45fc59193e86406bd68fc4ff5ae027862"} Nov 02 17:31:05.830965 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:05.830812 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"00b0092d-4e1f-4bb5-97a3-4e6e7e369d82","Type":"ContainerStarted","Data":"120f0c255be1d208c15f1b622b6134c63eb553aec0b82264e6dc5621ff508c8c"} Nov 02 17:31:05.872500 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:05.872453 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-684568bcb7-h8wc6" podStartSLOduration=1.8724354600000002 podStartE2EDuration="1.87243546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:31:05.871668574 +0000 UTC m=+83.567024046" watchObservedRunningTime="2026-11-02 17:31:05.87243546 +0000 UTC m=+83.567790933" Nov 02 17:31:05.939564 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:05.939519 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.9227003639999998 podStartE2EDuration="2.939506827s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.806463ms" isStatefulPod=false podCreationTimestamp="2026-11-02 17:31:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:31:05.937923193 +0000 UTC m=+83.633278678" watchObservedRunningTime="2026-11-02 17:31:05.939506827 +0000 UTC m=+83.634862300" Nov 02 17:31:06.438698 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:06.438664 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80554576-a567-4705-acd5-2b933615e02d" path="/var/lib/kubelet/pods/80554576-a567-4705-acd5-2b933615e02d/volumes" Nov 02 17:31:09.181464 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:31:09.181424 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:31:14.596012 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:14.595971 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-684568bcb7-h8wc6" Nov 02 17:31:14.596497 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:14.596097 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-684568bcb7-h8wc6" Nov 02 17:31:14.600663 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:14.600642 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-684568bcb7-h8wc6" Nov 02 17:31:14.866870 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:14.866794 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-684568bcb7-h8wc6" Nov 02 17:31:14.945252 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:14.945217 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58f656c7fb-bzfsh"] Nov 02 17:31:39.181839 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:31:39.181800 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:31:39.964702 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:39.964640 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-58f656c7fb-bzfsh" podUID="aaa8718c-561e-4164-aec5-ac3bba86706b" containerName="console" containerID="cri-o://63259ff3a373fae7fcba2e48fecefc140f56d0be535a7b92b052067f00ac3010" gracePeriod=15 Nov 02 17:31:40.203714 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:40.203691 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58f656c7fb-bzfsh" Nov 02 17:31:40.255421 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:40.255350 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaa8718c-561e-4164-aec5-ac3bba86706b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa8718c-561e-4164-aec5-ac3bba86706b-trusted-ca-bundle\") pod \"aaa8718c-561e-4164-aec5-ac3bba86706b\" (UID: \"aaa8718c-561e-4164-aec5-ac3bba86706b\") " Nov 02 17:31:40.255421 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:40.255415 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaa8718c-561e-4164-aec5-ac3bba86706b-tmp\" (UniqueName: \"kubernetes.io/empty-dir/aaa8718c-561e-4164-aec5-ac3bba86706b-tmp\") pod \"aaa8718c-561e-4164-aec5-ac3bba86706b\" (UID: \"aaa8718c-561e-4164-aec5-ac3bba86706b\") " Nov 02 17:31:40.255610 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:40.255446 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aaa8718c-561e-4164-aec5-ac3bba86706b-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aaa8718c-561e-4164-aec5-ac3bba86706b-console-oauth-config\") pod \"aaa8718c-561e-4164-aec5-ac3bba86706b\" (UID: \"aaa8718c-561e-4164-aec5-ac3bba86706b\") " Nov 02 17:31:40.255610 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:40.255483 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaa8718c-561e-4164-aec5-ac3bba86706b-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aaa8718c-561e-4164-aec5-ac3bba86706b-oauth-serving-cert\") pod \"aaa8718c-561e-4164-aec5-ac3bba86706b\" (UID: \"aaa8718c-561e-4164-aec5-ac3bba86706b\") " Nov 02 17:31:40.255610 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:40.255509 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aaa8718c-561e-4164-aec5-ac3bba86706b-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aaa8718c-561e-4164-aec5-ac3bba86706b-console-serving-cert\") pod \"aaa8718c-561e-4164-aec5-ac3bba86706b\" (UID: \"aaa8718c-561e-4164-aec5-ac3bba86706b\") " Nov 02 17:31:40.255770 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:40.255625 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaa8718c-561e-4164-aec5-ac3bba86706b-kube-api-access-g26n2\" (UniqueName: \"kubernetes.io/projected/aaa8718c-561e-4164-aec5-ac3bba86706b-kube-api-access-g26n2\") pod \"aaa8718c-561e-4164-aec5-ac3bba86706b\" (UID: \"aaa8718c-561e-4164-aec5-ac3bba86706b\") " Nov 02 17:31:40.255770 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:40.255668 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaa8718c-561e-4164-aec5-ac3bba86706b-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa8718c-561e-4164-aec5-ac3bba86706b-service-ca\") pod \"aaa8718c-561e-4164-aec5-ac3bba86706b\" (UID: \"aaa8718c-561e-4164-aec5-ac3bba86706b\") " Nov 02 17:31:40.255770 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:40.255693 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaa8718c-561e-4164-aec5-ac3bba86706b-tmp" pod "aaa8718c-561e-4164-aec5-ac3bba86706b" (UID: "aaa8718c-561e-4164-aec5-ac3bba86706b"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:31:40.255770 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:40.255716 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaa8718c-561e-4164-aec5-ac3bba86706b-console-config\" (UniqueName: \"kubernetes.io/configmap/aaa8718c-561e-4164-aec5-ac3bba86706b-console-config\") pod \"aaa8718c-561e-4164-aec5-ac3bba86706b\" (UID: \"aaa8718c-561e-4164-aec5-ac3bba86706b\") " Nov 02 17:31:40.255973 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:40.255780 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa8718c-561e-4164-aec5-ac3bba86706b-trusted-ca-bundle" pod "aaa8718c-561e-4164-aec5-ac3bba86706b" (UID: "aaa8718c-561e-4164-aec5-ac3bba86706b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:31:40.255973 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:40.255917 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa8718c-561e-4164-aec5-ac3bba86706b-oauth-serving-cert" pod "aaa8718c-561e-4164-aec5-ac3bba86706b" (UID: "aaa8718c-561e-4164-aec5-ac3bba86706b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:31:40.256080 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:40.255977 4471 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa8718c-561e-4164-aec5-ac3bba86706b-trusted-ca-bundle\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:31:40.256080 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:40.255996 4471 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aaa8718c-561e-4164-aec5-ac3bba86706b-tmp\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:31:40.256578 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:40.256321 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa8718c-561e-4164-aec5-ac3bba86706b-console-config" pod "aaa8718c-561e-4164-aec5-ac3bba86706b" (UID: "aaa8718c-561e-4164-aec5-ac3bba86706b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:31:40.256578 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:40.256570 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa8718c-561e-4164-aec5-ac3bba86706b-service-ca" pod "aaa8718c-561e-4164-aec5-ac3bba86706b" (UID: "aaa8718c-561e-4164-aec5-ac3bba86706b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:31:40.257940 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:40.257913 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaa8718c-561e-4164-aec5-ac3bba86706b-console-serving-cert" pod "aaa8718c-561e-4164-aec5-ac3bba86706b" (UID: "aaa8718c-561e-4164-aec5-ac3bba86706b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:31:40.258051 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:40.257986 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaa8718c-561e-4164-aec5-ac3bba86706b-console-oauth-config" pod "aaa8718c-561e-4164-aec5-ac3bba86706b" (UID: "aaa8718c-561e-4164-aec5-ac3bba86706b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:31:40.258163 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:40.258139 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaa8718c-561e-4164-aec5-ac3bba86706b-kube-api-access-g26n2" pod "aaa8718c-561e-4164-aec5-ac3bba86706b" (UID: "aaa8718c-561e-4164-aec5-ac3bba86706b"). InnerVolumeSpecName "kube-api-access-g26n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:31:40.356537 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:40.356487 4471 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aaa8718c-561e-4164-aec5-ac3bba86706b-console-oauth-config\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:31:40.356537 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:40.356531 4471 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aaa8718c-561e-4164-aec5-ac3bba86706b-oauth-serving-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:31:40.356537 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:40.356540 4471 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aaa8718c-561e-4164-aec5-ac3bba86706b-console-serving-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:31:40.356537 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:40.356549 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g26n2\" (UniqueName: \"kubernetes.io/projected/aaa8718c-561e-4164-aec5-ac3bba86706b-kube-api-access-g26n2\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:31:40.356537 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:40.356558 4471 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa8718c-561e-4164-aec5-ac3bba86706b-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:31:40.356846 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:40.356567 4471 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aaa8718c-561e-4164-aec5-ac3bba86706b-console-config\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:31:40.953713 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:40.953677 4471 generic.go:356] "Generic (PLEG): container finished" podID="aaa8718c-561e-4164-aec5-ac3bba86706b" containerID="63259ff3a373fae7fcba2e48fecefc140f56d0be535a7b92b052067f00ac3010" exitCode=0 Nov 02 17:31:40.953862 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:40.953726 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58f656c7fb-bzfsh" Nov 02 17:31:40.953862 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:40.953760 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58f656c7fb-bzfsh" event={"ID":"aaa8718c-561e-4164-aec5-ac3bba86706b","Type":"ContainerDied","Data":"63259ff3a373fae7fcba2e48fecefc140f56d0be535a7b92b052067f00ac3010"} Nov 02 17:31:40.953862 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:40.953813 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58f656c7fb-bzfsh" event={"ID":"aaa8718c-561e-4164-aec5-ac3bba86706b","Type":"ContainerDied","Data":"193d5a5151e3e9ee6f4c5c848b8f95ab7b1cb4f38d5e28a7fc0bfa89086648f3"} Nov 02 17:31:40.953862 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:40.953836 4471 scope.go:122] "RemoveContainer" containerID="63259ff3a373fae7fcba2e48fecefc140f56d0be535a7b92b052067f00ac3010" Nov 02 17:31:40.961538 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:40.961523 4471 scope.go:122] "RemoveContainer" containerID="63259ff3a373fae7fcba2e48fecefc140f56d0be535a7b92b052067f00ac3010" Nov 02 17:31:40.961778 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:31:40.961761 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63259ff3a373fae7fcba2e48fecefc140f56d0be535a7b92b052067f00ac3010\": container with ID starting with 63259ff3a373fae7fcba2e48fecefc140f56d0be535a7b92b052067f00ac3010 not found: ID does not exist" containerID="63259ff3a373fae7fcba2e48fecefc140f56d0be535a7b92b052067f00ac3010" Nov 02 17:31:40.961816 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:40.961788 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63259ff3a373fae7fcba2e48fecefc140f56d0be535a7b92b052067f00ac3010"} err="failed to get container status \"63259ff3a373fae7fcba2e48fecefc140f56d0be535a7b92b052067f00ac3010\": rpc error: code = NotFound desc = could not find container \"63259ff3a373fae7fcba2e48fecefc140f56d0be535a7b92b052067f00ac3010\": container with ID starting with 63259ff3a373fae7fcba2e48fecefc140f56d0be535a7b92b052067f00ac3010 not found: ID does not exist" Nov 02 17:31:41.013317 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:41.013286 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58f656c7fb-bzfsh"] Nov 02 17:31:41.028577 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:41.028551 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58f656c7fb-bzfsh"] Nov 02 17:31:42.437332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:31:42.437286 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaa8718c-561e-4164-aec5-ac3bba86706b" path="/var/lib/kubelet/pods/aaa8718c-561e-4164-aec5-ac3bba86706b/volumes" Nov 02 17:32:09.181266 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:32:09.181185 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:32:26.641709 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.641676 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm"] Nov 02 17:32:26.642152 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.641973 4471 cpu_manager.go:414] "containerMap: removing container" podUID="aaa8718c-561e-4164-aec5-ac3bba86706b" containerName="console" Nov 02 17:32:26.642152 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.641985 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaa8718c-561e-4164-aec5-ac3bba86706b" containerName="console" Nov 02 17:32:26.642152 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.642047 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaa8718c-561e-4164-aec5-ac3bba86706b" containerName="console" Nov 02 17:32:26.644666 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.644645 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" Nov 02 17:32:26.661602 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.661580 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 17:32:26.661747 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.661648 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 17:32:26.673653 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.673627 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 17:32:26.673801 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.673728 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 17:32:26.673801 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.673788 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 17:32:26.673929 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.673896 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dklz6\"" Nov 02 17:32:26.673998 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.673985 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 17:32:26.675046 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.675026 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm"] Nov 02 17:32:26.789209 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.789177 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-djbvm\" (UID: \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" Nov 02 17:32:26.789397 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.789221 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-djbvm\" (UID: \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" Nov 02 17:32:26.789397 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.789255 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-djbvm\" (UID: \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" Nov 02 17:32:26.789397 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.789367 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-djbvm\" (UID: \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" Nov 02 17:32:26.789397 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.789392 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-djbvm\" (UID: \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" Nov 02 17:32:26.789605 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.789421 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-djbvm\" (UID: \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" Nov 02 17:32:26.789605 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.789439 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ncnf\" (UniqueName: \"kubernetes.io/projected/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-kube-api-access-9ncnf\") pod \"istiod-openshift-gateway-7876bfcb74-djbvm\" (UID: \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" Nov 02 17:32:26.890831 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.890782 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-djbvm\" (UID: \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" Nov 02 17:32:26.891020 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.890840 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-djbvm\" (UID: \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" Nov 02 17:32:26.891020 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.890886 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-djbvm\" (UID: \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" Nov 02 17:32:26.891020 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.890916 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-djbvm\" (UID: \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" Nov 02 17:32:26.891020 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.890968 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-djbvm\" (UID: \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" Nov 02 17:32:26.891020 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.890993 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ncnf\" (UniqueName: \"kubernetes.io/projected/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-kube-api-access-9ncnf\") pod \"istiod-openshift-gateway-7876bfcb74-djbvm\" (UID: \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" Nov 02 17:32:26.891265 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.891036 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-djbvm\" (UID: \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" Nov 02 17:32:26.891522 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.891494 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-djbvm\" (UID: \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" Nov 02 17:32:26.893265 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.893209 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-djbvm\" (UID: \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" Nov 02 17:32:26.893422 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.893367 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-djbvm\" (UID: \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" Nov 02 17:32:26.893422 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.893385 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-djbvm\" (UID: \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" Nov 02 17:32:26.893516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.893426 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-djbvm\" (UID: \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" Nov 02 17:32:26.915023 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.915003 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-djbvm\" (UID: \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" Nov 02 17:32:26.915392 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.915373 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ncnf\" (UniqueName: \"kubernetes.io/projected/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-kube-api-access-9ncnf\") pod \"istiod-openshift-gateway-7876bfcb74-djbvm\" (UID: \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" Nov 02 17:32:26.956321 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:26.956278 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" Nov 02 17:32:27.263035 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:27.263008 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm"] Nov 02 17:32:27.263643 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:32:27.263613 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e8b6701_aa2e_4a0b_8e5e_a62ded370b1c.slice/crio-ad0bf20a61e413d5d16d6bc3399ab4d71c98f56d4624458d3130aab1008a911d WatchSource:0}: Error finding container ad0bf20a61e413d5d16d6bc3399ab4d71c98f56d4624458d3130aab1008a911d: Status 404 returned error can't find the container with id ad0bf20a61e413d5d16d6bc3399ab4d71c98f56d4624458d3130aab1008a911d Nov 02 17:32:28.111765 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:28.111644 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" event={"ID":"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c","Type":"ContainerStarted","Data":"ad0bf20a61e413d5d16d6bc3399ab4d71c98f56d4624458d3130aab1008a911d"} Nov 02 17:32:29.604951 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:29.604922 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:32:29.605210 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:29.604985 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:32:30.120885 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:30.120850 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" event={"ID":"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c","Type":"ContainerStarted","Data":"b7cab0c4c2953cf0a93676f6e09e241d05ce8585e8c5092d0726ffd9e608b26c"} Nov 02 17:32:30.121081 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:30.121025 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" Nov 02 17:32:30.122570 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:30.122543 4471 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-djbvm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 17:32:30.122692 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:30.122597 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" podUID="7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:32:30.191949 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:30.191899 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" podStartSLOduration=1.85334913 podStartE2EDuration="4.191886461s" totalImagesPullingTime="2.338537331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:32:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:30.179007024 +0000 UTC m=+167.874362498" watchObservedRunningTime="2026-11-02 17:32:30.191886461 +0000 UTC m=+167.887241933" Nov 02 17:32:31.124794 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:31.124766 4471 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-djbvm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 17:32:31.125157 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:31.124809 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" podUID="7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:32:34.125538 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:32:34.125507 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" Nov 02 17:32:39.181996 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:32:39.181952 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:33:09.181355 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:33:09.181295 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:33:24.489121 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:24.489080 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-js886"] Nov 02 17:33:24.495608 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:24.495583 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-js886" Nov 02 17:33:24.546651 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:24.546619 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 17:33:24.546651 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:24.546641 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 17:33:24.547735 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:24.547715 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-js886"] Nov 02 17:33:24.564297 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:24.564273 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 17:33:24.564438 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:24.564336 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 17:33:24.564525 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:24.564510 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 17:33:24.564597 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:24.564582 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-nv7lh\"" Nov 02 17:33:24.595944 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:24.595920 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a64f1879-c209-4dc5-8c2b-8c70f256a6c8-manifests\") pod \"opendatahub-operator-controller-manager-76f76b79c-js886\" (UID: \"a64f1879-c209-4dc5-8c2b-8c70f256a6c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-js886" Nov 02 17:33:24.596077 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:24.595954 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a64f1879-c209-4dc5-8c2b-8c70f256a6c8-metrics-cert\") pod \"opendatahub-operator-controller-manager-76f76b79c-js886\" (UID: \"a64f1879-c209-4dc5-8c2b-8c70f256a6c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-js886" Nov 02 17:33:24.596077 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:24.596052 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a64f1879-c209-4dc5-8c2b-8c70f256a6c8-tmp\") pod \"opendatahub-operator-controller-manager-76f76b79c-js886\" (UID: \"a64f1879-c209-4dc5-8c2b-8c70f256a6c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-js886" Nov 02 17:33:24.596158 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:24.596094 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a64f1879-c209-4dc5-8c2b-8c70f256a6c8-webhook-cert\") pod \"opendatahub-operator-controller-manager-76f76b79c-js886\" (UID: \"a64f1879-c209-4dc5-8c2b-8c70f256a6c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-js886" Nov 02 17:33:24.596192 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:24.596155 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwgdl\" (UniqueName: \"kubernetes.io/projected/a64f1879-c209-4dc5-8c2b-8c70f256a6c8-kube-api-access-wwgdl\") pod \"opendatahub-operator-controller-manager-76f76b79c-js886\" (UID: \"a64f1879-c209-4dc5-8c2b-8c70f256a6c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-js886" Nov 02 17:33:24.596257 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:24.596233 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a64f1879-c209-4dc5-8c2b-8c70f256a6c8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76f76b79c-js886\" (UID: \"a64f1879-c209-4dc5-8c2b-8c70f256a6c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-js886" Nov 02 17:33:24.696979 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:24.696951 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a64f1879-c209-4dc5-8c2b-8c70f256a6c8-manifests\") pod \"opendatahub-operator-controller-manager-76f76b79c-js886\" (UID: \"a64f1879-c209-4dc5-8c2b-8c70f256a6c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-js886" Nov 02 17:33:24.697138 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:24.696984 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a64f1879-c209-4dc5-8c2b-8c70f256a6c8-metrics-cert\") pod \"opendatahub-operator-controller-manager-76f76b79c-js886\" (UID: \"a64f1879-c209-4dc5-8c2b-8c70f256a6c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-js886" Nov 02 17:33:24.697138 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:24.697012 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a64f1879-c209-4dc5-8c2b-8c70f256a6c8-tmp\") pod \"opendatahub-operator-controller-manager-76f76b79c-js886\" (UID: \"a64f1879-c209-4dc5-8c2b-8c70f256a6c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-js886" Nov 02 17:33:24.697257 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:24.697133 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a64f1879-c209-4dc5-8c2b-8c70f256a6c8-webhook-cert\") pod \"opendatahub-operator-controller-manager-76f76b79c-js886\" (UID: \"a64f1879-c209-4dc5-8c2b-8c70f256a6c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-js886" Nov 02 17:33:24.697257 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:24.697189 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwgdl\" (UniqueName: \"kubernetes.io/projected/a64f1879-c209-4dc5-8c2b-8c70f256a6c8-kube-api-access-wwgdl\") pod \"opendatahub-operator-controller-manager-76f76b79c-js886\" (UID: \"a64f1879-c209-4dc5-8c2b-8c70f256a6c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-js886" Nov 02 17:33:24.697257 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:24.697225 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a64f1879-c209-4dc5-8c2b-8c70f256a6c8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76f76b79c-js886\" (UID: \"a64f1879-c209-4dc5-8c2b-8c70f256a6c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-js886" Nov 02 17:33:24.697526 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:24.697500 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a64f1879-c209-4dc5-8c2b-8c70f256a6c8-manifests\") pod \"opendatahub-operator-controller-manager-76f76b79c-js886\" (UID: \"a64f1879-c209-4dc5-8c2b-8c70f256a6c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-js886" Nov 02 17:33:24.699625 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:24.699605 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a64f1879-c209-4dc5-8c2b-8c70f256a6c8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76f76b79c-js886\" (UID: \"a64f1879-c209-4dc5-8c2b-8c70f256a6c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-js886" Nov 02 17:33:24.699729 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:24.699688 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a64f1879-c209-4dc5-8c2b-8c70f256a6c8-webhook-cert\") pod \"opendatahub-operator-controller-manager-76f76b79c-js886\" (UID: \"a64f1879-c209-4dc5-8c2b-8c70f256a6c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-js886" Nov 02 17:33:24.699782 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:24.699742 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a64f1879-c209-4dc5-8c2b-8c70f256a6c8-tmp\") pod \"opendatahub-operator-controller-manager-76f76b79c-js886\" (UID: \"a64f1879-c209-4dc5-8c2b-8c70f256a6c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-js886" Nov 02 17:33:24.699875 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:24.699856 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a64f1879-c209-4dc5-8c2b-8c70f256a6c8-metrics-cert\") pod \"opendatahub-operator-controller-manager-76f76b79c-js886\" (UID: \"a64f1879-c209-4dc5-8c2b-8c70f256a6c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-js886" Nov 02 17:33:24.732561 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:24.732540 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwgdl\" (UniqueName: \"kubernetes.io/projected/a64f1879-c209-4dc5-8c2b-8c70f256a6c8-kube-api-access-wwgdl\") pod \"opendatahub-operator-controller-manager-76f76b79c-js886\" (UID: \"a64f1879-c209-4dc5-8c2b-8c70f256a6c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-js886" Nov 02 17:33:24.808746 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:24.808684 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-js886" Nov 02 17:33:25.151944 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:33:25.151910 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda64f1879_c209_4dc5_8c2b_8c70f256a6c8.slice/crio-b0aac82cde5985d269737581400b7e1486ae9f019a44d95307dfa2f28b9cbeb9 WatchSource:0}: Error finding container b0aac82cde5985d269737581400b7e1486ae9f019a44d95307dfa2f28b9cbeb9: Status 404 returned error can't find the container with id b0aac82cde5985d269737581400b7e1486ae9f019a44d95307dfa2f28b9cbeb9 Nov 02 17:33:25.164191 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:25.164164 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-js886"] Nov 02 17:33:25.302932 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:25.302898 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-js886" event={"ID":"a64f1879-c209-4dc5-8c2b-8c70f256a6c8","Type":"ContainerStarted","Data":"b0aac82cde5985d269737581400b7e1486ae9f019a44d95307dfa2f28b9cbeb9"} Nov 02 17:33:28.316179 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:28.316139 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-js886" event={"ID":"a64f1879-c209-4dc5-8c2b-8c70f256a6c8","Type":"ContainerStarted","Data":"d6326818a41eb50ce2ca87e22b52671b80a8e18916919a0ca9ffb8d16e2a9bdc"} Nov 02 17:33:29.320996 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:29.320958 4471 generic.go:356] "Generic (PLEG): container finished" podID="a64f1879-c209-4dc5-8c2b-8c70f256a6c8" containerID="d6326818a41eb50ce2ca87e22b52671b80a8e18916919a0ca9ffb8d16e2a9bdc" exitCode=0 Nov 02 17:33:29.321506 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:29.321008 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-js886" event={"ID":"a64f1879-c209-4dc5-8c2b-8c70f256a6c8","Type":"ContainerDied","Data":"d6326818a41eb50ce2ca87e22b52671b80a8e18916919a0ca9ffb8d16e2a9bdc"} Nov 02 17:33:30.326175 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:30.326140 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-js886" event={"ID":"a64f1879-c209-4dc5-8c2b-8c70f256a6c8","Type":"ContainerStarted","Data":"6d46d27d518376270aa0f73a80e2a097b94191475e3ac34788a448e16f1c0336"} Nov 02 17:33:30.326610 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:30.326254 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-js886" Nov 02 17:33:30.447842 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:30.447789 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-js886" podStartSLOduration=2.863914868 podStartE2EDuration="6.447774832s" totalImagesPullingTime="3.154438775s" totalInitContainerRuntime="429.421189ms" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:30.433827026 +0000 UTC m=+228.129182553" watchObservedRunningTime="2026-11-02 17:33:30.447774832 +0000 UTC m=+228.143130304" Nov 02 17:33:39.181435 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:33:39.181397 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:33:41.331238 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:41.331205 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-js886" Nov 02 17:33:59.586675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:59.586633 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-78648788bf-gnws5"] Nov 02 17:33:59.589077 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:59.589060 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-78648788bf-gnws5" Nov 02 17:33:59.620078 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:59.620050 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-78648788bf-gnws5"] Nov 02 17:33:59.653409 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:59.653212 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 17:33:59.653409 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:59.653321 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-4679w\"" Nov 02 17:33:59.653783 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:59.653759 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 02 17:33:59.687427 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:59.687399 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7061bad7-2417-4dc7-a025-b1fcb9aec58d-tls-certs\") pod \"kube-auth-proxy-78648788bf-gnws5\" (UID: \"7061bad7-2417-4dc7-a025-b1fcb9aec58d\") " pod="openshift-ingress/kube-auth-proxy-78648788bf-gnws5" Nov 02 17:33:59.687546 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:59.687451 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6npzp\" (UniqueName: \"kubernetes.io/projected/7061bad7-2417-4dc7-a025-b1fcb9aec58d-kube-api-access-6npzp\") pod \"kube-auth-proxy-78648788bf-gnws5\" (UID: \"7061bad7-2417-4dc7-a025-b1fcb9aec58d\") " pod="openshift-ingress/kube-auth-proxy-78648788bf-gnws5" Nov 02 17:33:59.687546 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:59.687478 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7061bad7-2417-4dc7-a025-b1fcb9aec58d-tmp\") pod \"kube-auth-proxy-78648788bf-gnws5\" (UID: \"7061bad7-2417-4dc7-a025-b1fcb9aec58d\") " pod="openshift-ingress/kube-auth-proxy-78648788bf-gnws5" Nov 02 17:33:59.788764 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:59.788726 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6npzp\" (UniqueName: \"kubernetes.io/projected/7061bad7-2417-4dc7-a025-b1fcb9aec58d-kube-api-access-6npzp\") pod \"kube-auth-proxy-78648788bf-gnws5\" (UID: \"7061bad7-2417-4dc7-a025-b1fcb9aec58d\") " pod="openshift-ingress/kube-auth-proxy-78648788bf-gnws5" Nov 02 17:33:59.788764 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:59.788764 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7061bad7-2417-4dc7-a025-b1fcb9aec58d-tmp\") pod \"kube-auth-proxy-78648788bf-gnws5\" (UID: \"7061bad7-2417-4dc7-a025-b1fcb9aec58d\") " pod="openshift-ingress/kube-auth-proxy-78648788bf-gnws5" Nov 02 17:33:59.788994 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:59.788810 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7061bad7-2417-4dc7-a025-b1fcb9aec58d-tls-certs\") pod \"kube-auth-proxy-78648788bf-gnws5\" (UID: \"7061bad7-2417-4dc7-a025-b1fcb9aec58d\") " pod="openshift-ingress/kube-auth-proxy-78648788bf-gnws5" Nov 02 17:33:59.791326 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:59.791282 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7061bad7-2417-4dc7-a025-b1fcb9aec58d-tmp\") pod \"kube-auth-proxy-78648788bf-gnws5\" (UID: \"7061bad7-2417-4dc7-a025-b1fcb9aec58d\") " pod="openshift-ingress/kube-auth-proxy-78648788bf-gnws5" Nov 02 17:33:59.791468 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:59.791446 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7061bad7-2417-4dc7-a025-b1fcb9aec58d-tls-certs\") pod \"kube-auth-proxy-78648788bf-gnws5\" (UID: \"7061bad7-2417-4dc7-a025-b1fcb9aec58d\") " pod="openshift-ingress/kube-auth-proxy-78648788bf-gnws5" Nov 02 17:33:59.826413 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:59.826385 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6npzp\" (UniqueName: \"kubernetes.io/projected/7061bad7-2417-4dc7-a025-b1fcb9aec58d-kube-api-access-6npzp\") pod \"kube-auth-proxy-78648788bf-gnws5\" (UID: \"7061bad7-2417-4dc7-a025-b1fcb9aec58d\") " pod="openshift-ingress/kube-auth-proxy-78648788bf-gnws5" Nov 02 17:33:59.900475 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:33:59.900405 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-78648788bf-gnws5" Nov 02 17:34:00.201280 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:00.201255 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-78648788bf-gnws5"] Nov 02 17:34:00.203076 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:34:00.203047 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7061bad7_2417_4dc7_a025_b1fcb9aec58d.slice/crio-8a28f5b004669a6c2075e05d8d5a352a4b01f0cde8b5b5cdc3cf08d296bb1ae0 WatchSource:0}: Error finding container 8a28f5b004669a6c2075e05d8d5a352a4b01f0cde8b5b5cdc3cf08d296bb1ae0: Status 404 returned error can't find the container with id 8a28f5b004669a6c2075e05d8d5a352a4b01f0cde8b5b5cdc3cf08d296bb1ae0 Nov 02 17:34:00.437514 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:00.437484 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-78648788bf-gnws5" event={"ID":"7061bad7-2417-4dc7-a025-b1fcb9aec58d","Type":"ContainerStarted","Data":"8a28f5b004669a6c2075e05d8d5a352a4b01f0cde8b5b5cdc3cf08d296bb1ae0"} Nov 02 17:34:00.764648 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:00.764612 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:34:00.767482 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:00.767461 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-f6b9c" Nov 02 17:34:00.795351 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:00.795326 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:34:00.808496 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:00.808474 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 02 17:34:00.808614 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:00.808479 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 17:34:00.808614 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:00.808499 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 17:34:00.811105 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:00.811085 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-p5nzm\"" Nov 02 17:34:00.844579 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:00.844537 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:34:00.896672 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:00.896634 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwr64\" (UniqueName: \"kubernetes.io/projected/ab7e11d6-997a-4765-a1b7-85daa191d013-kube-api-access-dwr64\") pod \"limitador-limitador-7c9b87d574-f6b9c\" (UID: \"ab7e11d6-997a-4765-a1b7-85daa191d013\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-f6b9c" Nov 02 17:34:00.896835 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:00.896743 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ab7e11d6-997a-4765-a1b7-85daa191d013-config-file\") pod \"limitador-limitador-7c9b87d574-f6b9c\" (UID: \"ab7e11d6-997a-4765-a1b7-85daa191d013\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-f6b9c" Nov 02 17:34:00.997445 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:00.997406 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ab7e11d6-997a-4765-a1b7-85daa191d013-config-file\") pod \"limitador-limitador-7c9b87d574-f6b9c\" (UID: \"ab7e11d6-997a-4765-a1b7-85daa191d013\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-f6b9c" Nov 02 17:34:00.997640 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:00.997531 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwr64\" (UniqueName: \"kubernetes.io/projected/ab7e11d6-997a-4765-a1b7-85daa191d013-kube-api-access-dwr64\") pod \"limitador-limitador-7c9b87d574-f6b9c\" (UID: \"ab7e11d6-997a-4765-a1b7-85daa191d013\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-f6b9c" Nov 02 17:34:00.998678 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:00.998650 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ab7e11d6-997a-4765-a1b7-85daa191d013-config-file\") pod \"limitador-limitador-7c9b87d574-f6b9c\" (UID: \"ab7e11d6-997a-4765-a1b7-85daa191d013\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-f6b9c" Nov 02 17:34:01.024330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:01.024248 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwr64\" (UniqueName: \"kubernetes.io/projected/ab7e11d6-997a-4765-a1b7-85daa191d013-kube-api-access-dwr64\") pod \"limitador-limitador-7c9b87d574-f6b9c\" (UID: \"ab7e11d6-997a-4765-a1b7-85daa191d013\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-f6b9c" Nov 02 17:34:01.078998 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:01.078958 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-f6b9c" Nov 02 17:34:01.396423 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:01.396396 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:34:01.397375 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:34:01.397344 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab7e11d6_997a_4765_a1b7_85daa191d013.slice/crio-d12b963f6bcd846ca2512da9a8c948bb0ef6c2d7bb060db949885b09512d134c WatchSource:0}: Error finding container d12b963f6bcd846ca2512da9a8c948bb0ef6c2d7bb060db949885b09512d134c: Status 404 returned error can't find the container with id d12b963f6bcd846ca2512da9a8c948bb0ef6c2d7bb060db949885b09512d134c Nov 02 17:34:01.440788 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:01.440751 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-f6b9c" event={"ID":"ab7e11d6-997a-4765-a1b7-85daa191d013","Type":"ContainerStarted","Data":"d12b963f6bcd846ca2512da9a8c948bb0ef6c2d7bb060db949885b09512d134c"} Nov 02 17:34:03.450941 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:03.450904 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-78648788bf-gnws5" event={"ID":"7061bad7-2417-4dc7-a025-b1fcb9aec58d","Type":"ContainerStarted","Data":"2af3605115a63d68f8ad6ce8f10b712ed381b9c486a12ef2b7b72ecaa1316d41"} Nov 02 17:34:03.498668 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:03.498613 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-78648788bf-gnws5" podStartSLOduration=1.719809835 podStartE2EDuration="4.498595147s" totalImagesPullingTime="2.778785312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:34:03.49674354 +0000 UTC m=+261.192099012" watchObservedRunningTime="2026-11-02 17:34:03.498595147 +0000 UTC m=+261.193950619" Nov 02 17:34:04.456263 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:04.456230 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-8hfs4"] Nov 02 17:34:04.459746 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:04.459723 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-f6b9c" event={"ID":"ab7e11d6-997a-4765-a1b7-85daa191d013","Type":"ContainerStarted","Data":"8b21beceb9d7ae6702c7a39c501e69bc3cda5d8f3a623035da7e4faa2b50ba08"} Nov 02 17:34:04.459864 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:04.459801 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8hfs4" Nov 02 17:34:04.481428 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:04.481407 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-5zhk2\"" Nov 02 17:34:04.488523 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:04.488502 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-8hfs4"] Nov 02 17:34:04.493713 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:04.493694 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 17:34:04.494135 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:04.494124 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 17:34:04.520244 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:04.520221 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 02 17:34:04.521241 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:04.521201 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-f6b9c" podStartSLOduration=2.199719951 podStartE2EDuration="4.521188255s" totalImagesPullingTime="2.321468304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:34:04.520364457 +0000 UTC m=+262.215719924" watchObservedRunningTime="2026-11-02 17:34:04.521188255 +0000 UTC m=+262.216543727" Nov 02 17:34:04.627539 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:04.627503 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7092e32e-80ce-47db-acfc-57735ad62658-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-8hfs4\" (UID: \"7092e32e-80ce-47db-acfc-57735ad62658\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8hfs4" Nov 02 17:34:04.627705 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:04.627566 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7092e32e-80ce-47db-acfc-57735ad62658-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-8hfs4\" (UID: \"7092e32e-80ce-47db-acfc-57735ad62658\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8hfs4" Nov 02 17:34:04.627705 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:04.627700 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7092e32e-80ce-47db-acfc-57735ad62658-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-8hfs4\" (UID: \"7092e32e-80ce-47db-acfc-57735ad62658\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8hfs4" Nov 02 17:34:04.627817 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:04.627730 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z229\" (UniqueName: \"kubernetes.io/projected/7092e32e-80ce-47db-acfc-57735ad62658-kube-api-access-9z229\") pod \"kserve-module-controller-manager-7566cf5fc8-8hfs4\" (UID: \"7092e32e-80ce-47db-acfc-57735ad62658\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8hfs4" Nov 02 17:34:04.729204 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:04.729139 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7092e32e-80ce-47db-acfc-57735ad62658-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-8hfs4\" (UID: \"7092e32e-80ce-47db-acfc-57735ad62658\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8hfs4" Nov 02 17:34:04.729204 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:04.729176 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7092e32e-80ce-47db-acfc-57735ad62658-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-8hfs4\" (UID: \"7092e32e-80ce-47db-acfc-57735ad62658\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8hfs4" Nov 02 17:34:04.729204 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:04.729203 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7092e32e-80ce-47db-acfc-57735ad62658-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-8hfs4\" (UID: \"7092e32e-80ce-47db-acfc-57735ad62658\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8hfs4" Nov 02 17:34:04.729410 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:04.729348 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z229\" (UniqueName: \"kubernetes.io/projected/7092e32e-80ce-47db-acfc-57735ad62658-kube-api-access-9z229\") pod \"kserve-module-controller-manager-7566cf5fc8-8hfs4\" (UID: \"7092e32e-80ce-47db-acfc-57735ad62658\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8hfs4" Nov 02 17:34:04.729558 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:04.729542 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7092e32e-80ce-47db-acfc-57735ad62658-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-8hfs4\" (UID: \"7092e32e-80ce-47db-acfc-57735ad62658\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8hfs4" Nov 02 17:34:04.729603 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:04.729575 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7092e32e-80ce-47db-acfc-57735ad62658-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-8hfs4\" (UID: \"7092e32e-80ce-47db-acfc-57735ad62658\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8hfs4" Nov 02 17:34:04.730036 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:04.730018 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7092e32e-80ce-47db-acfc-57735ad62658-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-8hfs4\" (UID: \"7092e32e-80ce-47db-acfc-57735ad62658\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8hfs4" Nov 02 17:34:04.753301 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:04.753271 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z229\" (UniqueName: \"kubernetes.io/projected/7092e32e-80ce-47db-acfc-57735ad62658-kube-api-access-9z229\") pod \"kserve-module-controller-manager-7566cf5fc8-8hfs4\" (UID: \"7092e32e-80ce-47db-acfc-57735ad62658\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8hfs4" Nov 02 17:34:04.772133 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:04.772110 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8hfs4" Nov 02 17:34:05.084345 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:05.084315 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-8hfs4"] Nov 02 17:34:05.085295 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:34:05.085269 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7092e32e_80ce_47db_acfc_57735ad62658.slice/crio-a4a2e72f830f4a4905cedd6469d6a5cb365ce9c707ac0352852f97d678daa8d2 WatchSource:0}: Error finding container a4a2e72f830f4a4905cedd6469d6a5cb365ce9c707ac0352852f97d678daa8d2: Status 404 returned error can't find the container with id a4a2e72f830f4a4905cedd6469d6a5cb365ce9c707ac0352852f97d678daa8d2 Nov 02 17:34:05.461652 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:05.461564 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8hfs4" event={"ID":"7092e32e-80ce-47db-acfc-57735ad62658","Type":"ContainerStarted","Data":"a4a2e72f830f4a4905cedd6469d6a5cb365ce9c707ac0352852f97d678daa8d2"} Nov 02 17:34:05.461999 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:05.461653 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-f6b9c" Nov 02 17:34:07.471661 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:07.471631 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8hfs4" event={"ID":"7092e32e-80ce-47db-acfc-57735ad62658","Type":"ContainerStarted","Data":"57a1e0308744ff998f7a611e4ca2df992ce4aa19b38a37ad4bdf82ad459bf530"} Nov 02 17:34:07.472048 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:07.471744 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8hfs4" Nov 02 17:34:07.539119 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:07.539066 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8hfs4" podStartSLOduration=2.033848586 podStartE2EDuration="3.539048649s" totalImagesPullingTime="1.505200063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:34:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:34:07.53866847 +0000 UTC m=+265.234023942" watchObservedRunningTime="2026-11-02 17:34:07.539048649 +0000 UTC m=+265.234404123" Nov 02 17:34:09.181693 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:34:09.181651 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:34:16.466869 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:16.466839 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-f6b9c" Nov 02 17:34:28.475805 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:28.475777 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8hfs4" Nov 02 17:34:39.181999 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:34:39.181954 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:34:59.515971 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:59.515659 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sxvjv"] Nov 02 17:34:59.519240 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:59.519221 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sxvjv" Nov 02 17:34:59.555501 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:59.555471 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sxvjv"] Nov 02 17:34:59.664902 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:59.664868 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/43917fff-d5ed-4c3f-b4e4-b74d98f39e21-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sxvjv\" (UID: \"43917fff-d5ed-4c3f-b4e4-b74d98f39e21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sxvjv" Nov 02 17:34:59.664902 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:59.664904 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/43917fff-d5ed-4c3f-b4e4-b74d98f39e21-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sxvjv\" (UID: \"43917fff-d5ed-4c3f-b4e4-b74d98f39e21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sxvjv" Nov 02 17:34:59.665138 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:59.664933 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/43917fff-d5ed-4c3f-b4e4-b74d98f39e21-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sxvjv\" (UID: \"43917fff-d5ed-4c3f-b4e4-b74d98f39e21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sxvjv" Nov 02 17:34:59.665138 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:59.665004 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/43917fff-d5ed-4c3f-b4e4-b74d98f39e21-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sxvjv\" (UID: \"43917fff-d5ed-4c3f-b4e4-b74d98f39e21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sxvjv" Nov 02 17:34:59.665234 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:59.665147 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nktw\" (UniqueName: \"kubernetes.io/projected/43917fff-d5ed-4c3f-b4e4-b74d98f39e21-kube-api-access-8nktw\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sxvjv\" (UID: \"43917fff-d5ed-4c3f-b4e4-b74d98f39e21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sxvjv" Nov 02 17:34:59.665234 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:59.665182 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43917fff-d5ed-4c3f-b4e4-b74d98f39e21-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sxvjv\" (UID: \"43917fff-d5ed-4c3f-b4e4-b74d98f39e21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sxvjv" Nov 02 17:34:59.665234 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:59.665209 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/43917fff-d5ed-4c3f-b4e4-b74d98f39e21-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sxvjv\" (UID: \"43917fff-d5ed-4c3f-b4e4-b74d98f39e21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sxvjv" Nov 02 17:34:59.766433 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:59.766345 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/43917fff-d5ed-4c3f-b4e4-b74d98f39e21-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sxvjv\" (UID: \"43917fff-d5ed-4c3f-b4e4-b74d98f39e21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sxvjv" Nov 02 17:34:59.766433 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:59.766379 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/43917fff-d5ed-4c3f-b4e4-b74d98f39e21-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sxvjv\" (UID: \"43917fff-d5ed-4c3f-b4e4-b74d98f39e21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sxvjv" Nov 02 17:34:59.766433 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:59.766403 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/43917fff-d5ed-4c3f-b4e4-b74d98f39e21-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sxvjv\" (UID: \"43917fff-d5ed-4c3f-b4e4-b74d98f39e21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sxvjv" Nov 02 17:34:59.766713 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:59.766513 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/43917fff-d5ed-4c3f-b4e4-b74d98f39e21-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sxvjv\" (UID: \"43917fff-d5ed-4c3f-b4e4-b74d98f39e21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sxvjv" Nov 02 17:34:59.766713 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:59.766574 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nktw\" (UniqueName: \"kubernetes.io/projected/43917fff-d5ed-4c3f-b4e4-b74d98f39e21-kube-api-access-8nktw\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sxvjv\" (UID: \"43917fff-d5ed-4c3f-b4e4-b74d98f39e21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sxvjv" Nov 02 17:34:59.766713 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:59.766617 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43917fff-d5ed-4c3f-b4e4-b74d98f39e21-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sxvjv\" (UID: \"43917fff-d5ed-4c3f-b4e4-b74d98f39e21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sxvjv" Nov 02 17:34:59.766713 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:59.766656 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/43917fff-d5ed-4c3f-b4e4-b74d98f39e21-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sxvjv\" (UID: \"43917fff-d5ed-4c3f-b4e4-b74d98f39e21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sxvjv" Nov 02 17:34:59.767410 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:59.767382 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/43917fff-d5ed-4c3f-b4e4-b74d98f39e21-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sxvjv\" (UID: \"43917fff-d5ed-4c3f-b4e4-b74d98f39e21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sxvjv" Nov 02 17:34:59.768799 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:59.768773 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/43917fff-d5ed-4c3f-b4e4-b74d98f39e21-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sxvjv\" (UID: \"43917fff-d5ed-4c3f-b4e4-b74d98f39e21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sxvjv" Nov 02 17:34:59.768890 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:59.768811 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/43917fff-d5ed-4c3f-b4e4-b74d98f39e21-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sxvjv\" (UID: \"43917fff-d5ed-4c3f-b4e4-b74d98f39e21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sxvjv" Nov 02 17:34:59.768992 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:59.768971 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/43917fff-d5ed-4c3f-b4e4-b74d98f39e21-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sxvjv\" (UID: \"43917fff-d5ed-4c3f-b4e4-b74d98f39e21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sxvjv" Nov 02 17:34:59.769032 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:59.768987 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/43917fff-d5ed-4c3f-b4e4-b74d98f39e21-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sxvjv\" (UID: \"43917fff-d5ed-4c3f-b4e4-b74d98f39e21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sxvjv" Nov 02 17:34:59.793115 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:59.793087 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43917fff-d5ed-4c3f-b4e4-b74d98f39e21-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sxvjv\" (UID: \"43917fff-d5ed-4c3f-b4e4-b74d98f39e21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sxvjv" Nov 02 17:34:59.793253 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:59.793236 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nktw\" (UniqueName: \"kubernetes.io/projected/43917fff-d5ed-4c3f-b4e4-b74d98f39e21-kube-api-access-8nktw\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sxvjv\" (UID: \"43917fff-d5ed-4c3f-b4e4-b74d98f39e21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sxvjv" Nov 02 17:34:59.830147 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:34:59.830124 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sxvjv" Nov 02 17:35:00.151256 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:00.151224 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sxvjv"] Nov 02 17:35:00.151724 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:35:00.151700 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43917fff_d5ed_4c3f_b4e4_b74d98f39e21.slice/crio-2a51b3b71b9831b0d7df26f35ddfe6cdd52c2085c4bb754aac201199c5ac0189 WatchSource:0}: Error finding container 2a51b3b71b9831b0d7df26f35ddfe6cdd52c2085c4bb754aac201199c5ac0189: Status 404 returned error can't find the container with id 2a51b3b71b9831b0d7df26f35ddfe6cdd52c2085c4bb754aac201199c5ac0189 Nov 02 17:35:00.153782 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:00.153752 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:35:00.153867 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:00.153813 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:35:00.696337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:00.693025 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sxvjv" event={"ID":"43917fff-d5ed-4c3f-b4e4-b74d98f39e21","Type":"ContainerStarted","Data":"83ea5abe29f7eb908def9da009fac160f77361c0c2df2be9438c2b2a398ef5eb"} Nov 02 17:35:00.696337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:00.693090 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sxvjv" event={"ID":"43917fff-d5ed-4c3f-b4e4-b74d98f39e21","Type":"ContainerStarted","Data":"2a51b3b71b9831b0d7df26f35ddfe6cdd52c2085c4bb754aac201199c5ac0189"} Nov 02 17:35:00.696337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:00.693139 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sxvjv" Nov 02 17:35:00.750285 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:00.750219 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sxvjv" podStartSLOduration=1.750201944 podStartE2EDuration="1.750201944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:35:00.748899457 +0000 UTC m=+318.444254968" watchObservedRunningTime="2026-11-02 17:35:00.750201944 +0000 UTC m=+318.445557419" Nov 02 17:35:01.698863 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:01.698829 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sxvjv" Nov 02 17:35:01.796557 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:01.796529 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm"] Nov 02 17:35:01.796736 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:01.796714 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" podUID="7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c" containerName="discovery" containerID="cri-o://b7cab0c4c2953cf0a93676f6e09e241d05ce8585e8c5092d0726ffd9e608b26c" gracePeriod=30 Nov 02 17:35:02.039377 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:02.039355 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" Nov 02 17:35:02.191507 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:02.191471 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-istio-token\" (UniqueName: \"kubernetes.io/projected/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-istio-token\") pod \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\" (UID: \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\") " Nov 02 17:35:02.191681 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:02.191542 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-cacerts\" (UniqueName: \"kubernetes.io/secret/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-cacerts\") pod \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\" (UID: \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\") " Nov 02 17:35:02.191729 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:02.191709 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-istio-csr-ca-configmap\") pod \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\" (UID: \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\") " Nov 02 17:35:02.191804 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:02.191786 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-istio-csr-dns-cert\") pod \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\" (UID: \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\") " Nov 02 17:35:02.191865 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:02.191848 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-kube-api-access-9ncnf\" (UniqueName: \"kubernetes.io/projected/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-kube-api-access-9ncnf\") pod \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\" (UID: \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\") " Nov 02 17:35:02.191926 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:02.191889 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-istio-kubeconfig\") pod \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\" (UID: \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\") " Nov 02 17:35:02.191926 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:02.191922 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-local-certs\") pod \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\" (UID: \"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c\") " Nov 02 17:35:02.192226 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:02.192079 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-istio-csr-ca-configmap" pod "7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c" (UID: "7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:35:02.192226 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:02.192198 4471 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-istio-csr-ca-configmap\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:35:02.194292 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:02.194267 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-istio-csr-dns-cert" pod "7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c" (UID: "7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:35:02.194460 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:02.194268 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-local-certs" pod "7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c" (UID: "7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:35:02.194559 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:02.194520 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-cacerts" pod "7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c" (UID: "7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:35:02.194616 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:02.194559 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-istio-token" pod "7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c" (UID: "7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:35:02.194655 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:02.194604 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-kube-api-access-9ncnf" pod "7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c" (UID: "7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c"). InnerVolumeSpecName "kube-api-access-9ncnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:35:02.194749 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:02.194733 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-istio-kubeconfig" pod "7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c" (UID: "7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:35:02.293554 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:02.293527 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:35:02.293554 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:02.293550 4471 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-cacerts\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:35:02.293722 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:02.293560 4471 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-istio-csr-dns-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:35:02.293722 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:02.293570 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ncnf\" (UniqueName: \"kubernetes.io/projected/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-kube-api-access-9ncnf\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:35:02.293722 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:02.293580 4471 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-istio-kubeconfig\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:35:02.293722 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:02.293588 4471 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c-local-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:35:02.701975 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:02.701894 4471 generic.go:356] "Generic (PLEG): container finished" podID="7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c" containerID="b7cab0c4c2953cf0a93676f6e09e241d05ce8585e8c5092d0726ffd9e608b26c" exitCode=0 Nov 02 17:35:02.701975 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:02.701956 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" Nov 02 17:35:02.702474 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:02.701975 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" event={"ID":"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c","Type":"ContainerDied","Data":"b7cab0c4c2953cf0a93676f6e09e241d05ce8585e8c5092d0726ffd9e608b26c"} Nov 02 17:35:02.702474 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:02.702025 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm" event={"ID":"7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c","Type":"ContainerDied","Data":"ad0bf20a61e413d5d16d6bc3399ab4d71c98f56d4624458d3130aab1008a911d"} Nov 02 17:35:02.702474 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:02.702044 4471 scope.go:122] "RemoveContainer" containerID="b7cab0c4c2953cf0a93676f6e09e241d05ce8585e8c5092d0726ffd9e608b26c" Nov 02 17:35:02.710076 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:02.710059 4471 scope.go:122] "RemoveContainer" containerID="b7cab0c4c2953cf0a93676f6e09e241d05ce8585e8c5092d0726ffd9e608b26c" Nov 02 17:35:02.710369 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:35:02.710349 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7cab0c4c2953cf0a93676f6e09e241d05ce8585e8c5092d0726ffd9e608b26c\": container with ID starting with b7cab0c4c2953cf0a93676f6e09e241d05ce8585e8c5092d0726ffd9e608b26c not found: ID does not exist" containerID="b7cab0c4c2953cf0a93676f6e09e241d05ce8585e8c5092d0726ffd9e608b26c" Nov 02 17:35:02.710424 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:02.710378 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7cab0c4c2953cf0a93676f6e09e241d05ce8585e8c5092d0726ffd9e608b26c"} err="failed to get container status \"b7cab0c4c2953cf0a93676f6e09e241d05ce8585e8c5092d0726ffd9e608b26c\": rpc error: code = NotFound desc = could not find container \"b7cab0c4c2953cf0a93676f6e09e241d05ce8585e8c5092d0726ffd9e608b26c\": container with ID starting with b7cab0c4c2953cf0a93676f6e09e241d05ce8585e8c5092d0726ffd9e608b26c not found: ID does not exist" Nov 02 17:35:02.769203 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:02.769173 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm"] Nov 02 17:35:02.782169 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:02.782142 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-djbvm"] Nov 02 17:35:04.437755 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:04.437721 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c" path="/var/lib/kubelet/pods/7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c/volumes" Nov 02 17:35:09.181858 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:35:09.181818 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:35:20.411145 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:20.411109 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-6sxzc"] Nov 02 17:35:20.411556 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:20.411439 4471 cpu_manager.go:414] "containerMap: removing container" podUID="7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c" containerName="discovery" Nov 02 17:35:20.411556 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:20.411450 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c" containerName="discovery" Nov 02 17:35:20.411556 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:20.411507 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e8b6701-aa2e-4a0b-8e5e-a62ded370b1c" containerName="discovery" Nov 02 17:35:20.414421 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:20.414405 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6sxzc" Nov 02 17:35:20.434935 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:20.434915 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gml5d\"" Nov 02 17:35:20.443516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:20.443470 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f746fbbb-b3c7-46fa-8868-5209b00c0f13-maas-api-tls\") pod \"maas-api-6d669c7fcd-6sxzc\" (UID: \"f746fbbb-b3c7-46fa-8868-5209b00c0f13\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6sxzc" Nov 02 17:35:20.443622 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:20.443527 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f746fbbb-b3c7-46fa-8868-5209b00c0f13-openshift-service-ca\") pod \"maas-api-6d669c7fcd-6sxzc\" (UID: \"f746fbbb-b3c7-46fa-8868-5209b00c0f13\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6sxzc" Nov 02 17:35:20.443682 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:20.443641 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6k8s\" (UniqueName: \"kubernetes.io/projected/f746fbbb-b3c7-46fa-8868-5209b00c0f13-kube-api-access-r6k8s\") pod \"maas-api-6d669c7fcd-6sxzc\" (UID: \"f746fbbb-b3c7-46fa-8868-5209b00c0f13\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6sxzc" Nov 02 17:35:20.443734 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:20.443693 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f746fbbb-b3c7-46fa-8868-5209b00c0f13-metrics-certs\") pod \"maas-api-6d669c7fcd-6sxzc\" (UID: \"f746fbbb-b3c7-46fa-8868-5209b00c0f13\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6sxzc" Nov 02 17:35:20.447725 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:20.447696 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 17:35:20.447827 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:20.447743 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 17:35:20.447953 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:20.447936 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 17:35:20.448021 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:20.447993 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 17:35:20.449910 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:20.449884 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-6sxzc"] Nov 02 17:35:20.544350 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:20.544319 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f746fbbb-b3c7-46fa-8868-5209b00c0f13-maas-api-tls\") pod \"maas-api-6d669c7fcd-6sxzc\" (UID: \"f746fbbb-b3c7-46fa-8868-5209b00c0f13\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6sxzc" Nov 02 17:35:20.544515 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:20.544354 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f746fbbb-b3c7-46fa-8868-5209b00c0f13-openshift-service-ca\") pod \"maas-api-6d669c7fcd-6sxzc\" (UID: \"f746fbbb-b3c7-46fa-8868-5209b00c0f13\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6sxzc" Nov 02 17:35:20.544515 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:20.544384 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6k8s\" (UniqueName: \"kubernetes.io/projected/f746fbbb-b3c7-46fa-8868-5209b00c0f13-kube-api-access-r6k8s\") pod \"maas-api-6d669c7fcd-6sxzc\" (UID: \"f746fbbb-b3c7-46fa-8868-5209b00c0f13\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6sxzc" Nov 02 17:35:20.544515 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:20.544410 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f746fbbb-b3c7-46fa-8868-5209b00c0f13-metrics-certs\") pod \"maas-api-6d669c7fcd-6sxzc\" (UID: \"f746fbbb-b3c7-46fa-8868-5209b00c0f13\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6sxzc" Nov 02 17:35:20.545112 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:20.545089 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f746fbbb-b3c7-46fa-8868-5209b00c0f13-openshift-service-ca\") pod \"maas-api-6d669c7fcd-6sxzc\" (UID: \"f746fbbb-b3c7-46fa-8868-5209b00c0f13\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6sxzc" Nov 02 17:35:20.546805 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:20.546778 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f746fbbb-b3c7-46fa-8868-5209b00c0f13-maas-api-tls\") pod \"maas-api-6d669c7fcd-6sxzc\" (UID: \"f746fbbb-b3c7-46fa-8868-5209b00c0f13\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6sxzc" Nov 02 17:35:20.547041 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:20.546924 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f746fbbb-b3c7-46fa-8868-5209b00c0f13-metrics-certs\") pod \"maas-api-6d669c7fcd-6sxzc\" (UID: \"f746fbbb-b3c7-46fa-8868-5209b00c0f13\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6sxzc" Nov 02 17:35:20.570535 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:20.570512 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6k8s\" (UniqueName: \"kubernetes.io/projected/f746fbbb-b3c7-46fa-8868-5209b00c0f13-kube-api-access-r6k8s\") pod \"maas-api-6d669c7fcd-6sxzc\" (UID: \"f746fbbb-b3c7-46fa-8868-5209b00c0f13\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6sxzc" Nov 02 17:35:20.725550 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:20.725471 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6sxzc" Nov 02 17:35:21.026561 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:21.026531 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-6sxzc"] Nov 02 17:35:21.030479 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:35:21.030448 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf746fbbb_b3c7_46fa_8868_5209b00c0f13.slice/crio-3797039b1acc1b4475e0c820774258969c6c70c91a92fd9d27a39a973cc6f01e WatchSource:0}: Error finding container 3797039b1acc1b4475e0c820774258969c6c70c91a92fd9d27a39a973cc6f01e: Status 404 returned error can't find the container with id 3797039b1acc1b4475e0c820774258969c6c70c91a92fd9d27a39a973cc6f01e Nov 02 17:35:21.031877 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:21.031857 4471 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:35:21.788768 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:21.788728 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6sxzc" event={"ID":"f746fbbb-b3c7-46fa-8868-5209b00c0f13","Type":"ContainerStarted","Data":"3797039b1acc1b4475e0c820774258969c6c70c91a92fd9d27a39a973cc6f01e"} Nov 02 17:35:22.380552 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:22.380520 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:35:23.799089 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:23.799051 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6sxzc" event={"ID":"f746fbbb-b3c7-46fa-8868-5209b00c0f13","Type":"ContainerStarted","Data":"8786b84d8f8ee446985fab8fccc51c13148df75935dd727e997f2fbd77ba5222"} Nov 02 17:35:23.799477 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:23.799196 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6sxzc" Nov 02 17:35:23.843364 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:23.843288 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6sxzc" podStartSLOduration=1.551809668 podStartE2EDuration="3.843271356s" totalImagesPullingTime="2.291461688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:35:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:35:23.84286884 +0000 UTC m=+341.538224314" watchObservedRunningTime="2026-11-02 17:35:23.843271356 +0000 UTC m=+341.538626825" Nov 02 17:35:29.808921 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:29.808891 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6sxzc" Nov 02 17:35:30.860996 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:30.860964 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-667c5f9d54-tq47s"] Nov 02 17:35:30.864056 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:30.864038 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-667c5f9d54-tq47s" Nov 02 17:35:30.885651 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:30.885618 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-667c5f9d54-tq47s"] Nov 02 17:35:30.925751 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:30.925725 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cccx\" (UniqueName: \"kubernetes.io/projected/d0158b72-5fdc-42dc-929d-d1e3663359f2-kube-api-access-4cccx\") pod \"maas-api-667c5f9d54-tq47s\" (UID: \"d0158b72-5fdc-42dc-929d-d1e3663359f2\") " pod="odh-ai-gateway-infra/maas-api-667c5f9d54-tq47s" Nov 02 17:35:30.925910 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:30.925761 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0158b72-5fdc-42dc-929d-d1e3663359f2-metrics-certs\") pod \"maas-api-667c5f9d54-tq47s\" (UID: \"d0158b72-5fdc-42dc-929d-d1e3663359f2\") " pod="odh-ai-gateway-infra/maas-api-667c5f9d54-tq47s" Nov 02 17:35:30.925910 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:30.925861 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0158b72-5fdc-42dc-929d-d1e3663359f2-maas-api-tls\") pod \"maas-api-667c5f9d54-tq47s\" (UID: \"d0158b72-5fdc-42dc-929d-d1e3663359f2\") " pod="odh-ai-gateway-infra/maas-api-667c5f9d54-tq47s" Nov 02 17:35:30.925988 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:30.925912 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0158b72-5fdc-42dc-929d-d1e3663359f2-openshift-service-ca\") pod \"maas-api-667c5f9d54-tq47s\" (UID: \"d0158b72-5fdc-42dc-929d-d1e3663359f2\") " pod="odh-ai-gateway-infra/maas-api-667c5f9d54-tq47s" Nov 02 17:35:31.027099 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:31.027064 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0158b72-5fdc-42dc-929d-d1e3663359f2-maas-api-tls\") pod \"maas-api-667c5f9d54-tq47s\" (UID: \"d0158b72-5fdc-42dc-929d-d1e3663359f2\") " pod="odh-ai-gateway-infra/maas-api-667c5f9d54-tq47s" Nov 02 17:35:31.027335 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:31.027110 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0158b72-5fdc-42dc-929d-d1e3663359f2-openshift-service-ca\") pod \"maas-api-667c5f9d54-tq47s\" (UID: \"d0158b72-5fdc-42dc-929d-d1e3663359f2\") " pod="odh-ai-gateway-infra/maas-api-667c5f9d54-tq47s" Nov 02 17:35:31.027335 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:31.027141 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cccx\" (UniqueName: \"kubernetes.io/projected/d0158b72-5fdc-42dc-929d-d1e3663359f2-kube-api-access-4cccx\") pod \"maas-api-667c5f9d54-tq47s\" (UID: \"d0158b72-5fdc-42dc-929d-d1e3663359f2\") " pod="odh-ai-gateway-infra/maas-api-667c5f9d54-tq47s" Nov 02 17:35:31.027335 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:31.027175 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0158b72-5fdc-42dc-929d-d1e3663359f2-metrics-certs\") pod \"maas-api-667c5f9d54-tq47s\" (UID: \"d0158b72-5fdc-42dc-929d-d1e3663359f2\") " pod="odh-ai-gateway-infra/maas-api-667c5f9d54-tq47s" Nov 02 17:35:31.027875 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:31.027852 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0158b72-5fdc-42dc-929d-d1e3663359f2-openshift-service-ca\") pod \"maas-api-667c5f9d54-tq47s\" (UID: \"d0158b72-5fdc-42dc-929d-d1e3663359f2\") " pod="odh-ai-gateway-infra/maas-api-667c5f9d54-tq47s" Nov 02 17:35:31.029580 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:31.029557 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0158b72-5fdc-42dc-929d-d1e3663359f2-maas-api-tls\") pod \"maas-api-667c5f9d54-tq47s\" (UID: \"d0158b72-5fdc-42dc-929d-d1e3663359f2\") " pod="odh-ai-gateway-infra/maas-api-667c5f9d54-tq47s" Nov 02 17:35:31.029687 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:31.029598 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0158b72-5fdc-42dc-929d-d1e3663359f2-metrics-certs\") pod \"maas-api-667c5f9d54-tq47s\" (UID: \"d0158b72-5fdc-42dc-929d-d1e3663359f2\") " pod="odh-ai-gateway-infra/maas-api-667c5f9d54-tq47s" Nov 02 17:35:31.051102 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:31.051073 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cccx\" (UniqueName: \"kubernetes.io/projected/d0158b72-5fdc-42dc-929d-d1e3663359f2-kube-api-access-4cccx\") pod \"maas-api-667c5f9d54-tq47s\" (UID: \"d0158b72-5fdc-42dc-929d-d1e3663359f2\") " pod="odh-ai-gateway-infra/maas-api-667c5f9d54-tq47s" Nov 02 17:35:31.176075 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:31.175994 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-667c5f9d54-tq47s" Nov 02 17:35:31.471962 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:31.471935 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-667c5f9d54-tq47s"] Nov 02 17:35:31.472661 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:35:31.472635 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0158b72_5fdc_42dc_929d_d1e3663359f2.slice/crio-4cd46e3ff7d71da3a9ed63b2ee7ec8bfaca96d74e9595ef7cbeec983432c2ad9 WatchSource:0}: Error finding container 4cd46e3ff7d71da3a9ed63b2ee7ec8bfaca96d74e9595ef7cbeec983432c2ad9: Status 404 returned error can't find the container with id 4cd46e3ff7d71da3a9ed63b2ee7ec8bfaca96d74e9595ef7cbeec983432c2ad9 Nov 02 17:35:31.832249 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:31.832214 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-667c5f9d54-tq47s" event={"ID":"d0158b72-5fdc-42dc-929d-d1e3663359f2","Type":"ContainerStarted","Data":"4cd46e3ff7d71da3a9ed63b2ee7ec8bfaca96d74e9595ef7cbeec983432c2ad9"} Nov 02 17:35:32.837163 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:32.837123 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-667c5f9d54-tq47s" event={"ID":"d0158b72-5fdc-42dc-929d-d1e3663359f2","Type":"ContainerStarted","Data":"735e891456133727f3f7ab93b450fc96b15918a47815ee8d4ae733e86186b367"} Nov 02 17:35:32.837617 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:32.837236 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-667c5f9d54-tq47s" Nov 02 17:35:32.882328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:32.882262 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-667c5f9d54-tq47s" podStartSLOduration=2.57183799 podStartE2EDuration="2.882248526s" totalImagesPullingTime="310.410536ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:35:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:35:32.881160505 +0000 UTC m=+350.576515995" watchObservedRunningTime="2026-11-02 17:35:32.882248526 +0000 UTC m=+350.577603997" Nov 02 17:35:38.847118 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:38.847086 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-667c5f9d54-tq47s" Nov 02 17:35:38.922014 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:38.921978 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-6sxzc"] Nov 02 17:35:38.922261 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:38.922207 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6sxzc" podUID="f746fbbb-b3c7-46fa-8868-5209b00c0f13" containerName="maas-api" containerID="cri-o://8786b84d8f8ee446985fab8fccc51c13148df75935dd727e997f2fbd77ba5222" gracePeriod=30 Nov 02 17:35:39.169586 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:39.169555 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6sxzc" Nov 02 17:35:39.187387 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:35:39.187355 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:35:39.293143 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:39.293113 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f746fbbb-b3c7-46fa-8868-5209b00c0f13-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f746fbbb-b3c7-46fa-8868-5209b00c0f13-openshift-service-ca\") pod \"f746fbbb-b3c7-46fa-8868-5209b00c0f13\" (UID: \"f746fbbb-b3c7-46fa-8868-5209b00c0f13\") " Nov 02 17:35:39.293348 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:39.293163 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f746fbbb-b3c7-46fa-8868-5209b00c0f13-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f746fbbb-b3c7-46fa-8868-5209b00c0f13-metrics-certs\") pod \"f746fbbb-b3c7-46fa-8868-5209b00c0f13\" (UID: \"f746fbbb-b3c7-46fa-8868-5209b00c0f13\") " Nov 02 17:35:39.293348 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:39.293225 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f746fbbb-b3c7-46fa-8868-5209b00c0f13-kube-api-access-r6k8s\" (UniqueName: \"kubernetes.io/projected/f746fbbb-b3c7-46fa-8868-5209b00c0f13-kube-api-access-r6k8s\") pod \"f746fbbb-b3c7-46fa-8868-5209b00c0f13\" (UID: \"f746fbbb-b3c7-46fa-8868-5209b00c0f13\") " Nov 02 17:35:39.293348 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:39.293250 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f746fbbb-b3c7-46fa-8868-5209b00c0f13-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f746fbbb-b3c7-46fa-8868-5209b00c0f13-maas-api-tls\") pod \"f746fbbb-b3c7-46fa-8868-5209b00c0f13\" (UID: \"f746fbbb-b3c7-46fa-8868-5209b00c0f13\") " Nov 02 17:35:39.293573 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:39.293546 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f746fbbb-b3c7-46fa-8868-5209b00c0f13-openshift-service-ca" pod "f746fbbb-b3c7-46fa-8868-5209b00c0f13" (UID: "f746fbbb-b3c7-46fa-8868-5209b00c0f13"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:35:39.295550 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:39.295526 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f746fbbb-b3c7-46fa-8868-5209b00c0f13-metrics-certs" pod "f746fbbb-b3c7-46fa-8868-5209b00c0f13" (UID: "f746fbbb-b3c7-46fa-8868-5209b00c0f13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:35:39.295550 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:39.295532 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f746fbbb-b3c7-46fa-8868-5209b00c0f13-kube-api-access-r6k8s" pod "f746fbbb-b3c7-46fa-8868-5209b00c0f13" (UID: "f746fbbb-b3c7-46fa-8868-5209b00c0f13"). InnerVolumeSpecName "kube-api-access-r6k8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:35:39.295672 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:39.295546 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f746fbbb-b3c7-46fa-8868-5209b00c0f13-maas-api-tls" pod "f746fbbb-b3c7-46fa-8868-5209b00c0f13" (UID: "f746fbbb-b3c7-46fa-8868-5209b00c0f13"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:35:39.393905 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:39.393873 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6k8s\" (UniqueName: \"kubernetes.io/projected/f746fbbb-b3c7-46fa-8868-5209b00c0f13-kube-api-access-r6k8s\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:35:39.393905 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:39.393901 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f746fbbb-b3c7-46fa-8868-5209b00c0f13-maas-api-tls\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:35:39.393905 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:39.393910 4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f746fbbb-b3c7-46fa-8868-5209b00c0f13-openshift-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:35:39.394092 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:39.393919 4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f746fbbb-b3c7-46fa-8868-5209b00c0f13-metrics-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:35:39.866971 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:39.866936 4471 generic.go:356] "Generic (PLEG): container finished" podID="f746fbbb-b3c7-46fa-8868-5209b00c0f13" containerID="8786b84d8f8ee446985fab8fccc51c13148df75935dd727e997f2fbd77ba5222" exitCode=0 Nov 02 17:35:39.867415 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:39.866982 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6sxzc" Nov 02 17:35:39.867415 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:39.867022 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6sxzc" event={"ID":"f746fbbb-b3c7-46fa-8868-5209b00c0f13","Type":"ContainerDied","Data":"8786b84d8f8ee446985fab8fccc51c13148df75935dd727e997f2fbd77ba5222"} Nov 02 17:35:39.867415 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:39.867078 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6sxzc" event={"ID":"f746fbbb-b3c7-46fa-8868-5209b00c0f13","Type":"ContainerDied","Data":"3797039b1acc1b4475e0c820774258969c6c70c91a92fd9d27a39a973cc6f01e"} Nov 02 17:35:39.867415 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:39.867101 4471 scope.go:122] "RemoveContainer" containerID="8786b84d8f8ee446985fab8fccc51c13148df75935dd727e997f2fbd77ba5222" Nov 02 17:35:39.875809 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:39.875793 4471 scope.go:122] "RemoveContainer" containerID="8786b84d8f8ee446985fab8fccc51c13148df75935dd727e997f2fbd77ba5222" Nov 02 17:35:39.876104 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:35:39.876086 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8786b84d8f8ee446985fab8fccc51c13148df75935dd727e997f2fbd77ba5222\": container with ID starting with 8786b84d8f8ee446985fab8fccc51c13148df75935dd727e997f2fbd77ba5222 not found: ID does not exist" containerID="8786b84d8f8ee446985fab8fccc51c13148df75935dd727e997f2fbd77ba5222" Nov 02 17:35:39.876152 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:39.876112 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8786b84d8f8ee446985fab8fccc51c13148df75935dd727e997f2fbd77ba5222"} err="failed to get container status \"8786b84d8f8ee446985fab8fccc51c13148df75935dd727e997f2fbd77ba5222\": rpc error: code = NotFound desc = could not find container \"8786b84d8f8ee446985fab8fccc51c13148df75935dd727e997f2fbd77ba5222\": container with ID starting with 8786b84d8f8ee446985fab8fccc51c13148df75935dd727e997f2fbd77ba5222 not found: ID does not exist" Nov 02 17:35:39.919950 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:39.919926 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-6sxzc"] Nov 02 17:35:39.932773 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:39.932752 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-6sxzc"] Nov 02 17:35:40.439133 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:35:40.439099 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f746fbbb-b3c7-46fa-8868-5209b00c0f13" path="/var/lib/kubelet/pods/f746fbbb-b3c7-46fa-8868-5209b00c0f13/volumes" Nov 02 17:36:09.181898 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:36:09.181858 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:36:39.181565 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:36:39.181525 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:36:48.126529 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:48.126496 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf"] Nov 02 17:36:48.127014 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:48.126999 4471 cpu_manager.go:414] "containerMap: removing container" podUID="f746fbbb-b3c7-46fa-8868-5209b00c0f13" containerName="maas-api" Nov 02 17:36:48.127062 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:48.127017 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f746fbbb-b3c7-46fa-8868-5209b00c0f13" containerName="maas-api" Nov 02 17:36:48.127123 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:48.127112 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="f746fbbb-b3c7-46fa-8868-5209b00c0f13" containerName="maas-api" Nov 02 17:36:48.129712 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:48.129687 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf" Nov 02 17:36:48.157742 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:48.157715 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-f7th6\"" Nov 02 17:36:48.157742 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:48.157733 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 17:36:48.157742 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:48.157723 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 02 17:36:48.157933 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:48.157755 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 17:36:48.160105 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:48.160084 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf"] Nov 02 17:36:48.234020 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:48.233991 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad9d1c12-a624-47ee-80b2-db72091ec1f6-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v25sf\" (UID: \"ad9d1c12-a624-47ee-80b2-db72091ec1f6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf" Nov 02 17:36:48.234161 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:48.234029 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad9d1c12-a624-47ee-80b2-db72091ec1f6-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v25sf\" (UID: \"ad9d1c12-a624-47ee-80b2-db72091ec1f6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf" Nov 02 17:36:48.234161 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:48.234145 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lv65\" (UniqueName: \"kubernetes.io/projected/ad9d1c12-a624-47ee-80b2-db72091ec1f6-kube-api-access-6lv65\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v25sf\" (UID: \"ad9d1c12-a624-47ee-80b2-db72091ec1f6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf" Nov 02 17:36:48.234289 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:48.234178 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad9d1c12-a624-47ee-80b2-db72091ec1f6-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v25sf\" (UID: \"ad9d1c12-a624-47ee-80b2-db72091ec1f6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf" Nov 02 17:36:48.234289 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:48.234198 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad9d1c12-a624-47ee-80b2-db72091ec1f6-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v25sf\" (UID: \"ad9d1c12-a624-47ee-80b2-db72091ec1f6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf" Nov 02 17:36:48.234289 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:48.234230 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad9d1c12-a624-47ee-80b2-db72091ec1f6-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v25sf\" (UID: \"ad9d1c12-a624-47ee-80b2-db72091ec1f6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf" Nov 02 17:36:48.335366 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:48.335335 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad9d1c12-a624-47ee-80b2-db72091ec1f6-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v25sf\" (UID: \"ad9d1c12-a624-47ee-80b2-db72091ec1f6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf" Nov 02 17:36:48.335529 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:48.335384 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad9d1c12-a624-47ee-80b2-db72091ec1f6-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v25sf\" (UID: \"ad9d1c12-a624-47ee-80b2-db72091ec1f6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf" Nov 02 17:36:48.335529 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:48.335406 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad9d1c12-a624-47ee-80b2-db72091ec1f6-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v25sf\" (UID: \"ad9d1c12-a624-47ee-80b2-db72091ec1f6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf" Nov 02 17:36:48.335529 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:48.335459 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lv65\" (UniqueName: \"kubernetes.io/projected/ad9d1c12-a624-47ee-80b2-db72091ec1f6-kube-api-access-6lv65\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v25sf\" (UID: \"ad9d1c12-a624-47ee-80b2-db72091ec1f6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf" Nov 02 17:36:48.335529 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:48.335483 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad9d1c12-a624-47ee-80b2-db72091ec1f6-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v25sf\" (UID: \"ad9d1c12-a624-47ee-80b2-db72091ec1f6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf" Nov 02 17:36:48.335529 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:48.335510 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad9d1c12-a624-47ee-80b2-db72091ec1f6-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v25sf\" (UID: \"ad9d1c12-a624-47ee-80b2-db72091ec1f6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf" Nov 02 17:36:48.335842 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:48.335807 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad9d1c12-a624-47ee-80b2-db72091ec1f6-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v25sf\" (UID: \"ad9d1c12-a624-47ee-80b2-db72091ec1f6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf" Nov 02 17:36:48.335951 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:48.335917 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad9d1c12-a624-47ee-80b2-db72091ec1f6-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v25sf\" (UID: \"ad9d1c12-a624-47ee-80b2-db72091ec1f6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf" Nov 02 17:36:48.336005 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:48.335917 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad9d1c12-a624-47ee-80b2-db72091ec1f6-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v25sf\" (UID: \"ad9d1c12-a624-47ee-80b2-db72091ec1f6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf" Nov 02 17:36:48.337738 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:48.337715 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad9d1c12-a624-47ee-80b2-db72091ec1f6-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v25sf\" (UID: \"ad9d1c12-a624-47ee-80b2-db72091ec1f6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf" Nov 02 17:36:48.338156 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:48.338138 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad9d1c12-a624-47ee-80b2-db72091ec1f6-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v25sf\" (UID: \"ad9d1c12-a624-47ee-80b2-db72091ec1f6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf" Nov 02 17:36:48.363343 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:48.363322 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lv65\" (UniqueName: \"kubernetes.io/projected/ad9d1c12-a624-47ee-80b2-db72091ec1f6-kube-api-access-6lv65\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v25sf\" (UID: \"ad9d1c12-a624-47ee-80b2-db72091ec1f6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf" Nov 02 17:36:48.441167 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:48.441103 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf" Nov 02 17:36:48.734021 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:48.733990 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf"] Nov 02 17:36:48.734588 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:36:48.734562 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad9d1c12_a624_47ee_80b2_db72091ec1f6.slice/crio-019752d4fcfa25ee096e4da560cabce7536272229139497253243c9283785be6 WatchSource:0}: Error finding container 019752d4fcfa25ee096e4da560cabce7536272229139497253243c9283785be6: Status 404 returned error can't find the container with id 019752d4fcfa25ee096e4da560cabce7536272229139497253243c9283785be6 Nov 02 17:36:49.138941 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:49.138905 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf" event={"ID":"ad9d1c12-a624-47ee-80b2-db72091ec1f6","Type":"ContainerStarted","Data":"019752d4fcfa25ee096e4da560cabce7536272229139497253243c9283785be6"} Nov 02 17:36:49.283940 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:49.283898 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:36:50.144268 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:50.144190 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf" event={"ID":"ad9d1c12-a624-47ee-80b2-db72091ec1f6","Type":"ContainerStarted","Data":"49c6cc3bccb780974e9be82f70b6b864adcb66b341c9cfe7f8f461cefa6026ba"} Nov 02 17:36:50.189132 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:50.189082 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf" podStartSLOduration=1.137995935 podStartE2EDuration="2.189069631s" totalImagesPullingTime="1.051073696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:36:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:36:50.188692903 +0000 UTC m=+427.884048378" watchObservedRunningTime="2026-11-02 17:36:50.189069631 +0000 UTC m=+427.884425101" Nov 02 17:36:50.441869 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:50.441797 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf" Nov 02 17:36:52.348496 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:52.348463 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66"] Nov 02 17:36:52.351355 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:52.351332 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66" Nov 02 17:36:52.369583 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:52.369564 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 02 17:36:52.383003 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:52.382982 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66"] Nov 02 17:36:52.470587 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:52.470559 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26722ed8-2267-4729-8872-7fbf9a222c87-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66\" (UID: \"26722ed8-2267-4729-8872-7fbf9a222c87\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66" Nov 02 17:36:52.470587 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:52.470591 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26722ed8-2267-4729-8872-7fbf9a222c87-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66\" (UID: \"26722ed8-2267-4729-8872-7fbf9a222c87\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66" Nov 02 17:36:52.470806 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:52.470625 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26722ed8-2267-4729-8872-7fbf9a222c87-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66\" (UID: \"26722ed8-2267-4729-8872-7fbf9a222c87\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66" Nov 02 17:36:52.470806 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:52.470740 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26722ed8-2267-4729-8872-7fbf9a222c87-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66\" (UID: \"26722ed8-2267-4729-8872-7fbf9a222c87\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66" Nov 02 17:36:52.470806 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:52.470782 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26722ed8-2267-4729-8872-7fbf9a222c87-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66\" (UID: \"26722ed8-2267-4729-8872-7fbf9a222c87\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66" Nov 02 17:36:52.470806 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:52.470801 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbwp4\" (UniqueName: \"kubernetes.io/projected/26722ed8-2267-4729-8872-7fbf9a222c87-kube-api-access-wbwp4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66\" (UID: \"26722ed8-2267-4729-8872-7fbf9a222c87\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66" Nov 02 17:36:52.571868 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:52.571837 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26722ed8-2267-4729-8872-7fbf9a222c87-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66\" (UID: \"26722ed8-2267-4729-8872-7fbf9a222c87\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66" Nov 02 17:36:52.571868 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:52.571869 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26722ed8-2267-4729-8872-7fbf9a222c87-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66\" (UID: \"26722ed8-2267-4729-8872-7fbf9a222c87\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66" Nov 02 17:36:52.572085 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:52.571899 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26722ed8-2267-4729-8872-7fbf9a222c87-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66\" (UID: \"26722ed8-2267-4729-8872-7fbf9a222c87\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66" Nov 02 17:36:52.572085 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:52.571956 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26722ed8-2267-4729-8872-7fbf9a222c87-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66\" (UID: \"26722ed8-2267-4729-8872-7fbf9a222c87\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66" Nov 02 17:36:52.572085 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:52.572004 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26722ed8-2267-4729-8872-7fbf9a222c87-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66\" (UID: \"26722ed8-2267-4729-8872-7fbf9a222c87\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66" Nov 02 17:36:52.572085 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:52.572029 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbwp4\" (UniqueName: \"kubernetes.io/projected/26722ed8-2267-4729-8872-7fbf9a222c87-kube-api-access-wbwp4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66\" (UID: \"26722ed8-2267-4729-8872-7fbf9a222c87\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66" Nov 02 17:36:52.572292 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:52.572266 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26722ed8-2267-4729-8872-7fbf9a222c87-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66\" (UID: \"26722ed8-2267-4729-8872-7fbf9a222c87\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66" Nov 02 17:36:52.572446 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:52.572423 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26722ed8-2267-4729-8872-7fbf9a222c87-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66\" (UID: \"26722ed8-2267-4729-8872-7fbf9a222c87\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66" Nov 02 17:36:52.572446 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:52.572437 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26722ed8-2267-4729-8872-7fbf9a222c87-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66\" (UID: \"26722ed8-2267-4729-8872-7fbf9a222c87\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66" Nov 02 17:36:52.574324 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:52.574286 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26722ed8-2267-4729-8872-7fbf9a222c87-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66\" (UID: \"26722ed8-2267-4729-8872-7fbf9a222c87\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66" Nov 02 17:36:52.574558 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:52.574540 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26722ed8-2267-4729-8872-7fbf9a222c87-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66\" (UID: \"26722ed8-2267-4729-8872-7fbf9a222c87\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66" Nov 02 17:36:52.599666 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:52.599611 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbwp4\" (UniqueName: \"kubernetes.io/projected/26722ed8-2267-4729-8872-7fbf9a222c87-kube-api-access-wbwp4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66\" (UID: \"26722ed8-2267-4729-8872-7fbf9a222c87\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66" Nov 02 17:36:52.663578 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:52.663552 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66" Nov 02 17:36:52.958954 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:36:52.958926 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26722ed8_2267_4729_8872_7fbf9a222c87.slice/crio-52c13275c03155e7b547bd6c5333ab6dbfd4314352d48da566ddf2bce8c0feb6 WatchSource:0}: Error finding container 52c13275c03155e7b547bd6c5333ab6dbfd4314352d48da566ddf2bce8c0feb6: Status 404 returned error can't find the container with id 52c13275c03155e7b547bd6c5333ab6dbfd4314352d48da566ddf2bce8c0feb6 Nov 02 17:36:52.959395 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:52.959372 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66"] Nov 02 17:36:53.158662 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:53.158633 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66" event={"ID":"26722ed8-2267-4729-8872-7fbf9a222c87","Type":"ContainerStarted","Data":"52c13275c03155e7b547bd6c5333ab6dbfd4314352d48da566ddf2bce8c0feb6"} Nov 02 17:36:54.164729 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:54.164690 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66" event={"ID":"26722ed8-2267-4729-8872-7fbf9a222c87","Type":"ContainerStarted","Data":"e4140148ee1c190dffa724cc55d5da2c8b470f1d8ca69ab9e77da7a959ae2a66"} Nov 02 17:36:54.209131 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:54.209079 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66" podStartSLOduration=1.983430402 podStartE2EDuration="2.209065542s" totalImagesPullingTime="225.63514ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:36:54.20853416 +0000 UTC m=+431.903889659" watchObservedRunningTime="2026-11-02 17:36:54.209065542 +0000 UTC m=+431.904421017" Nov 02 17:36:54.311974 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:54.311940 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-j2l7j"] Nov 02 17:36:54.315578 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:54.315560 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-j2l7j" Nov 02 17:36:54.334466 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:54.334440 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 02 17:36:54.335949 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:54.335925 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-j2l7j"] Nov 02 17:36:54.357457 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:54.357426 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:36:54.387233 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:54.387207 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b966a58b-3f29-4f26-a84e-89999a77de25-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-j2l7j\" (UID: \"b966a58b-3f29-4f26-a84e-89999a77de25\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j2l7j" Nov 02 17:36:54.387389 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:54.387255 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b966a58b-3f29-4f26-a84e-89999a77de25-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-j2l7j\" (UID: \"b966a58b-3f29-4f26-a84e-89999a77de25\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j2l7j" Nov 02 17:36:54.387389 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:54.387279 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b966a58b-3f29-4f26-a84e-89999a77de25-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-j2l7j\" (UID: \"b966a58b-3f29-4f26-a84e-89999a77de25\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j2l7j" Nov 02 17:36:54.387491 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:54.387447 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqph2\" (UniqueName: \"kubernetes.io/projected/b966a58b-3f29-4f26-a84e-89999a77de25-kube-api-access-vqph2\") pod \"echo-simulator-kserve-64b6f8f96c-j2l7j\" (UID: \"b966a58b-3f29-4f26-a84e-89999a77de25\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j2l7j" Nov 02 17:36:54.387491 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:54.387476 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b966a58b-3f29-4f26-a84e-89999a77de25-home\") pod \"echo-simulator-kserve-64b6f8f96c-j2l7j\" (UID: \"b966a58b-3f29-4f26-a84e-89999a77de25\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j2l7j" Nov 02 17:36:54.387580 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:54.387494 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b966a58b-3f29-4f26-a84e-89999a77de25-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-j2l7j\" (UID: \"b966a58b-3f29-4f26-a84e-89999a77de25\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j2l7j" Nov 02 17:36:54.488716 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:54.488645 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqph2\" (UniqueName: \"kubernetes.io/projected/b966a58b-3f29-4f26-a84e-89999a77de25-kube-api-access-vqph2\") pod \"echo-simulator-kserve-64b6f8f96c-j2l7j\" (UID: \"b966a58b-3f29-4f26-a84e-89999a77de25\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j2l7j" Nov 02 17:36:54.488716 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:54.488680 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b966a58b-3f29-4f26-a84e-89999a77de25-home\") pod \"echo-simulator-kserve-64b6f8f96c-j2l7j\" (UID: \"b966a58b-3f29-4f26-a84e-89999a77de25\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j2l7j" Nov 02 17:36:54.488716 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:54.488698 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b966a58b-3f29-4f26-a84e-89999a77de25-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-j2l7j\" (UID: \"b966a58b-3f29-4f26-a84e-89999a77de25\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j2l7j" Nov 02 17:36:54.488953 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:54.488723 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b966a58b-3f29-4f26-a84e-89999a77de25-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-j2l7j\" (UID: \"b966a58b-3f29-4f26-a84e-89999a77de25\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j2l7j" Nov 02 17:36:54.488953 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:54.488885 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b966a58b-3f29-4f26-a84e-89999a77de25-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-j2l7j\" (UID: \"b966a58b-3f29-4f26-a84e-89999a77de25\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j2l7j" Nov 02 17:36:54.488953 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:54.488937 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b966a58b-3f29-4f26-a84e-89999a77de25-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-j2l7j\" (UID: \"b966a58b-3f29-4f26-a84e-89999a77de25\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j2l7j" Nov 02 17:36:54.489154 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:54.489128 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b966a58b-3f29-4f26-a84e-89999a77de25-home\") pod \"echo-simulator-kserve-64b6f8f96c-j2l7j\" (UID: \"b966a58b-3f29-4f26-a84e-89999a77de25\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j2l7j" Nov 02 17:36:54.489211 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:54.489164 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b966a58b-3f29-4f26-a84e-89999a77de25-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-j2l7j\" (UID: \"b966a58b-3f29-4f26-a84e-89999a77de25\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j2l7j" Nov 02 17:36:54.489654 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:54.489630 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b966a58b-3f29-4f26-a84e-89999a77de25-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-j2l7j\" (UID: \"b966a58b-3f29-4f26-a84e-89999a77de25\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j2l7j" Nov 02 17:36:54.491279 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:54.491257 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b966a58b-3f29-4f26-a84e-89999a77de25-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-j2l7j\" (UID: \"b966a58b-3f29-4f26-a84e-89999a77de25\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j2l7j" Nov 02 17:36:54.491499 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:54.491482 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b966a58b-3f29-4f26-a84e-89999a77de25-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-j2l7j\" (UID: \"b966a58b-3f29-4f26-a84e-89999a77de25\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j2l7j" Nov 02 17:36:54.512426 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:54.512404 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqph2\" (UniqueName: \"kubernetes.io/projected/b966a58b-3f29-4f26-a84e-89999a77de25-kube-api-access-vqph2\") pod \"echo-simulator-kserve-64b6f8f96c-j2l7j\" (UID: \"b966a58b-3f29-4f26-a84e-89999a77de25\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j2l7j" Nov 02 17:36:54.629368 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:54.629344 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-j2l7j" Nov 02 17:36:54.664457 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:54.664432 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66" Nov 02 17:36:54.920881 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:54.920853 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-j2l7j"] Nov 02 17:36:54.921428 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:36:54.921401 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb966a58b_3f29_4f26_a84e_89999a77de25.slice/crio-e31fbf41a42462993a34dff58ad3bc825e0dcea22d62a83648c567a842276eae WatchSource:0}: Error finding container e31fbf41a42462993a34dff58ad3bc825e0dcea22d62a83648c567a842276eae: Status 404 returned error can't find the container with id e31fbf41a42462993a34dff58ad3bc825e0dcea22d62a83648c567a842276eae Nov 02 17:36:55.170601 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:55.170524 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-j2l7j" event={"ID":"b966a58b-3f29-4f26-a84e-89999a77de25","Type":"ContainerStarted","Data":"e31fbf41a42462993a34dff58ad3bc825e0dcea22d62a83648c567a842276eae"} Nov 02 17:36:57.181160 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:57.181123 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-j2l7j" event={"ID":"b966a58b-3f29-4f26-a84e-89999a77de25","Type":"ContainerStarted","Data":"191cf0f57d4237b601bf2ce77a81e9129af995f230420ac0b6468ea598a871d2"} Nov 02 17:36:57.224653 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:57.224607 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-j2l7j" podStartSLOduration=1.7858460630000002 podStartE2EDuration="3.224592644s" totalImagesPullingTime="1.438746581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:36:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:36:57.224192569 +0000 UTC m=+434.919548072" watchObservedRunningTime="2026-11-02 17:36:57.224592644 +0000 UTC m=+434.919948117" Nov 02 17:36:57.371883 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:57.371847 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h"] Nov 02 17:36:57.376059 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:57.376037 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 17:36:57.394995 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:57.394969 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h"] Nov 02 17:36:57.395648 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:57.395631 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 02 17:36:57.515358 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:57.515328 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c5408ad-4b7b-47c0-854a-6e6c3a723302-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h\" (UID: \"5c5408ad-4b7b-47c0-854a-6e6c3a723302\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 17:36:57.515523 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:57.515440 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njzpw\" (UniqueName: \"kubernetes.io/projected/5c5408ad-4b7b-47c0-854a-6e6c3a723302-kube-api-access-njzpw\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h\" (UID: \"5c5408ad-4b7b-47c0-854a-6e6c3a723302\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 17:36:57.515523 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:57.515487 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c5408ad-4b7b-47c0-854a-6e6c3a723302-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h\" (UID: \"5c5408ad-4b7b-47c0-854a-6e6c3a723302\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 17:36:57.515621 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:57.515581 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c5408ad-4b7b-47c0-854a-6e6c3a723302-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h\" (UID: \"5c5408ad-4b7b-47c0-854a-6e6c3a723302\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 17:36:57.515621 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:57.515610 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c5408ad-4b7b-47c0-854a-6e6c3a723302-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h\" (UID: \"5c5408ad-4b7b-47c0-854a-6e6c3a723302\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 17:36:57.515713 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:57.515639 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c5408ad-4b7b-47c0-854a-6e6c3a723302-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h\" (UID: \"5c5408ad-4b7b-47c0-854a-6e6c3a723302\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 17:36:57.616890 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:57.616854 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c5408ad-4b7b-47c0-854a-6e6c3a723302-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h\" (UID: \"5c5408ad-4b7b-47c0-854a-6e6c3a723302\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 17:36:57.617063 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:57.616909 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c5408ad-4b7b-47c0-854a-6e6c3a723302-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h\" (UID: \"5c5408ad-4b7b-47c0-854a-6e6c3a723302\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 17:36:57.617063 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:57.616941 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njzpw\" (UniqueName: \"kubernetes.io/projected/5c5408ad-4b7b-47c0-854a-6e6c3a723302-kube-api-access-njzpw\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h\" (UID: \"5c5408ad-4b7b-47c0-854a-6e6c3a723302\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 17:36:57.617063 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:57.616963 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c5408ad-4b7b-47c0-854a-6e6c3a723302-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h\" (UID: \"5c5408ad-4b7b-47c0-854a-6e6c3a723302\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 17:36:57.617063 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:57.616981 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c5408ad-4b7b-47c0-854a-6e6c3a723302-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h\" (UID: \"5c5408ad-4b7b-47c0-854a-6e6c3a723302\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 17:36:57.617063 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:57.617000 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c5408ad-4b7b-47c0-854a-6e6c3a723302-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h\" (UID: \"5c5408ad-4b7b-47c0-854a-6e6c3a723302\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 17:36:57.617415 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:57.617388 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c5408ad-4b7b-47c0-854a-6e6c3a723302-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h\" (UID: \"5c5408ad-4b7b-47c0-854a-6e6c3a723302\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 17:36:57.617415 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:57.617404 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c5408ad-4b7b-47c0-854a-6e6c3a723302-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h\" (UID: \"5c5408ad-4b7b-47c0-854a-6e6c3a723302\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 17:36:57.617500 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:57.617448 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c5408ad-4b7b-47c0-854a-6e6c3a723302-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h\" (UID: \"5c5408ad-4b7b-47c0-854a-6e6c3a723302\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 17:36:57.619190 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:57.619173 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c5408ad-4b7b-47c0-854a-6e6c3a723302-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h\" (UID: \"5c5408ad-4b7b-47c0-854a-6e6c3a723302\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 17:36:57.619523 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:57.619506 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c5408ad-4b7b-47c0-854a-6e6c3a723302-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h\" (UID: \"5c5408ad-4b7b-47c0-854a-6e6c3a723302\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 17:36:57.630321 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:57.630283 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-j2l7j" Nov 02 17:36:57.660788 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:57.660765 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njzpw\" (UniqueName: \"kubernetes.io/projected/5c5408ad-4b7b-47c0-854a-6e6c3a723302-kube-api-access-njzpw\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h\" (UID: \"5c5408ad-4b7b-47c0-854a-6e6c3a723302\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 17:36:57.702882 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:57.702857 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 17:36:57.998810 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:57.998781 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h"] Nov 02 17:36:57.999378 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:36:57.999353 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c5408ad_4b7b_47c0_854a_6e6c3a723302.slice/crio-6634f237298d0a8e662e1f812acfb823eceb6bdb8de20407c7c102806eee8863 WatchSource:0}: Error finding container 6634f237298d0a8e662e1f812acfb823eceb6bdb8de20407c7c102806eee8863: Status 404 returned error can't find the container with id 6634f237298d0a8e662e1f812acfb823eceb6bdb8de20407c7c102806eee8863 Nov 02 17:36:58.186146 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:58.186114 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" event={"ID":"5c5408ad-4b7b-47c0-854a-6e6c3a723302","Type":"ContainerStarted","Data":"6634f237298d0a8e662e1f812acfb823eceb6bdb8de20407c7c102806eee8863"} Nov 02 17:36:58.441742 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:58.441710 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf" Nov 02 17:36:58.454194 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:58.454173 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf" Nov 02 17:36:59.191212 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:59.191162 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" event={"ID":"5c5408ad-4b7b-47c0-854a-6e6c3a723302","Type":"ContainerStarted","Data":"744fb020e51c628e24a93549f3dcb5179b53b51d0c53eb437b839776db2456f2"} Nov 02 17:36:59.202600 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:59.202575 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf" Nov 02 17:36:59.235868 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:59.235823 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podStartSLOduration=2.024168707 podStartE2EDuration="2.235810976s" totalImagesPullingTime="211.642269ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:36:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:36:59.234496149 +0000 UTC m=+436.929851623" watchObservedRunningTime="2026-11-02 17:36:59.235810976 +0000 UTC m=+436.931166448" Nov 02 17:36:59.703698 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:36:59.703669 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 17:37:00.075892 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:00.075858 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:37:01.879587 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:01.879558 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-drrc4"] Nov 02 17:37:01.883070 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:01.883049 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-drrc4" Nov 02 17:37:01.912198 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:01.912178 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 02 17:37:01.913813 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:01.913789 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-drrc4"] Nov 02 17:37:01.958230 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:01.958200 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/880a572e-4d2f-4f0c-9f31-1f40f37c1723-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-drrc4\" (UID: \"880a572e-4d2f-4f0c-9f31-1f40f37c1723\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-drrc4" Nov 02 17:37:01.958369 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:01.958239 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/880a572e-4d2f-4f0c-9f31-1f40f37c1723-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-drrc4\" (UID: \"880a572e-4d2f-4f0c-9f31-1f40f37c1723\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-drrc4" Nov 02 17:37:01.958422 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:01.958363 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drjb2\" (UniqueName: \"kubernetes.io/projected/880a572e-4d2f-4f0c-9f31-1f40f37c1723-kube-api-access-drjb2\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-drrc4\" (UID: \"880a572e-4d2f-4f0c-9f31-1f40f37c1723\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-drrc4" Nov 02 17:37:01.958422 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:01.958411 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/880a572e-4d2f-4f0c-9f31-1f40f37c1723-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-drrc4\" (UID: \"880a572e-4d2f-4f0c-9f31-1f40f37c1723\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-drrc4" Nov 02 17:37:01.958505 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:01.958492 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/880a572e-4d2f-4f0c-9f31-1f40f37c1723-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-drrc4\" (UID: \"880a572e-4d2f-4f0c-9f31-1f40f37c1723\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-drrc4" Nov 02 17:37:01.958540 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:01.958519 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/880a572e-4d2f-4f0c-9f31-1f40f37c1723-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-drrc4\" (UID: \"880a572e-4d2f-4f0c-9f31-1f40f37c1723\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-drrc4" Nov 02 17:37:02.059454 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:02.059408 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/880a572e-4d2f-4f0c-9f31-1f40f37c1723-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-drrc4\" (UID: \"880a572e-4d2f-4f0c-9f31-1f40f37c1723\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-drrc4" Nov 02 17:37:02.059632 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:02.059506 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drjb2\" (UniqueName: \"kubernetes.io/projected/880a572e-4d2f-4f0c-9f31-1f40f37c1723-kube-api-access-drjb2\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-drrc4\" (UID: \"880a572e-4d2f-4f0c-9f31-1f40f37c1723\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-drrc4" Nov 02 17:37:02.059632 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:02.059550 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/880a572e-4d2f-4f0c-9f31-1f40f37c1723-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-drrc4\" (UID: \"880a572e-4d2f-4f0c-9f31-1f40f37c1723\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-drrc4" Nov 02 17:37:02.059632 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:02.059616 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/880a572e-4d2f-4f0c-9f31-1f40f37c1723-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-drrc4\" (UID: \"880a572e-4d2f-4f0c-9f31-1f40f37c1723\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-drrc4" Nov 02 17:37:02.059785 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:02.059654 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/880a572e-4d2f-4f0c-9f31-1f40f37c1723-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-drrc4\" (UID: \"880a572e-4d2f-4f0c-9f31-1f40f37c1723\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-drrc4" Nov 02 17:37:02.059785 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:02.059692 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/880a572e-4d2f-4f0c-9f31-1f40f37c1723-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-drrc4\" (UID: \"880a572e-4d2f-4f0c-9f31-1f40f37c1723\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-drrc4" Nov 02 17:37:02.059927 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:02.059896 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/880a572e-4d2f-4f0c-9f31-1f40f37c1723-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-drrc4\" (UID: \"880a572e-4d2f-4f0c-9f31-1f40f37c1723\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-drrc4" Nov 02 17:37:02.059968 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:02.059896 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/880a572e-4d2f-4f0c-9f31-1f40f37c1723-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-drrc4\" (UID: \"880a572e-4d2f-4f0c-9f31-1f40f37c1723\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-drrc4" Nov 02 17:37:02.060008 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:02.059977 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/880a572e-4d2f-4f0c-9f31-1f40f37c1723-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-drrc4\" (UID: \"880a572e-4d2f-4f0c-9f31-1f40f37c1723\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-drrc4" Nov 02 17:37:02.061935 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:02.061907 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/880a572e-4d2f-4f0c-9f31-1f40f37c1723-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-drrc4\" (UID: \"880a572e-4d2f-4f0c-9f31-1f40f37c1723\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-drrc4" Nov 02 17:37:02.062188 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:02.062167 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/880a572e-4d2f-4f0c-9f31-1f40f37c1723-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-drrc4\" (UID: \"880a572e-4d2f-4f0c-9f31-1f40f37c1723\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-drrc4" Nov 02 17:37:02.083888 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:02.083857 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drjb2\" (UniqueName: \"kubernetes.io/projected/880a572e-4d2f-4f0c-9f31-1f40f37c1723-kube-api-access-drjb2\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-drrc4\" (UID: \"880a572e-4d2f-4f0c-9f31-1f40f37c1723\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-drrc4" Nov 02 17:37:02.196169 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:02.196094 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-drrc4" Nov 02 17:37:02.489993 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:02.489961 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-drrc4"] Nov 02 17:37:02.490331 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:37:02.490286 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod880a572e_4d2f_4f0c_9f31_1f40f37c1723.slice/crio-5ba2334bae3f5e7c242fc25e5156363d6e1de0624f4724e7552f6da498bf8652 WatchSource:0}: Error finding container 5ba2334bae3f5e7c242fc25e5156363d6e1de0624f4724e7552f6da498bf8652: Status 404 returned error can't find the container with id 5ba2334bae3f5e7c242fc25e5156363d6e1de0624f4724e7552f6da498bf8652 Nov 02 17:37:02.664580 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:02.664550 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66" Nov 02 17:37:02.683091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:02.683065 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66" Nov 02 17:37:03.209235 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:03.209200 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-drrc4" event={"ID":"880a572e-4d2f-4f0c-9f31-1f40f37c1723","Type":"ContainerStarted","Data":"4bd0bab936bff260ae7cd482d65c673416404dc5f8a68e1c88ac07992083cdbd"} Nov 02 17:37:03.209839 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:03.209256 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-drrc4" event={"ID":"880a572e-4d2f-4f0c-9f31-1f40f37c1723","Type":"ContainerStarted","Data":"5ba2334bae3f5e7c242fc25e5156363d6e1de0624f4724e7552f6da498bf8652"} Nov 02 17:37:03.220440 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:03.220416 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66" Nov 02 17:37:03.253414 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:03.253370 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-drrc4" podStartSLOduration=2.038403706 podStartE2EDuration="2.253356771s" totalImagesPullingTime="214.953065ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:37:03.252640698 +0000 UTC m=+440.947996171" watchObservedRunningTime="2026-11-02 17:37:03.253356771 +0000 UTC m=+440.948712244" Nov 02 17:37:04.197031 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:04.196995 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-drrc4" Nov 02 17:37:04.630323 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:04.630265 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-j2l7j" Nov 02 17:37:04.642554 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:04.642529 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-j2l7j" Nov 02 17:37:05.228702 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:05.228675 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-j2l7j" Nov 02 17:37:07.458423 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:07.458390 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:37:07.703636 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:07.703603 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 17:37:07.715970 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:07.715906 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 17:37:08.241427 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:08.241398 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 17:37:09.181286 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:37:09.181245 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:37:12.196900 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:12.196864 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-drrc4" Nov 02 17:37:12.209384 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:12.209358 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-drrc4" Nov 02 17:37:12.257818 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:12.257792 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-drrc4" Nov 02 17:37:22.319678 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:22.319642 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-2p9rr"] Nov 02 17:37:22.323262 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:22.323246 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-2p9rr" Nov 02 17:37:22.343897 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:22.343876 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-6nwlt\"" Nov 02 17:37:22.344027 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:22.343918 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 02 17:37:22.345168 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:22.345141 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-2p9rr"] Nov 02 17:37:22.347394 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:22.347371 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz9w5\" (UniqueName: \"kubernetes.io/projected/85011b6f-dbf8-472b-bf28-f13cc9d99de4-kube-api-access-hz9w5\") pod \"payload-pre-processing-7b98bb678c-2p9rr\" (UID: \"85011b6f-dbf8-472b-bf28-f13cc9d99de4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-2p9rr" Nov 02 17:37:22.347507 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:22.347423 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85011b6f-dbf8-472b-bf28-f13cc9d99de4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-2p9rr\" (UID: \"85011b6f-dbf8-472b-bf28-f13cc9d99de4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-2p9rr" Nov 02 17:37:22.422634 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:22.422607 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-rjwjd"] Nov 02 17:37:22.425894 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:22.425878 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-rjwjd" Nov 02 17:37:22.446784 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:22.446756 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-rjwjd"] Nov 02 17:37:22.448182 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:22.448157 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz9w5\" (UniqueName: \"kubernetes.io/projected/85011b6f-dbf8-472b-bf28-f13cc9d99de4-kube-api-access-hz9w5\") pod \"payload-pre-processing-7b98bb678c-2p9rr\" (UID: \"85011b6f-dbf8-472b-bf28-f13cc9d99de4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-2p9rr" Nov 02 17:37:22.448332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:22.448197 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-446pp\" (UniqueName: \"kubernetes.io/projected/91ff9502-0ff4-43a1-8f78-b5d8c8fe683e-kube-api-access-446pp\") pod \"payload-processing-55fbbbf9dc-rjwjd\" (UID: \"91ff9502-0ff4-43a1-8f78-b5d8c8fe683e\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rjwjd" Nov 02 17:37:22.448332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:22.448232 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91ff9502-0ff4-43a1-8f78-b5d8c8fe683e-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-rjwjd\" (UID: \"91ff9502-0ff4-43a1-8f78-b5d8c8fe683e\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rjwjd" Nov 02 17:37:22.448332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:22.448261 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85011b6f-dbf8-472b-bf28-f13cc9d99de4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-2p9rr\" (UID: \"85011b6f-dbf8-472b-bf28-f13cc9d99de4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-2p9rr" Nov 02 17:37:22.448870 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:22.448849 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85011b6f-dbf8-472b-bf28-f13cc9d99de4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-2p9rr\" (UID: \"85011b6f-dbf8-472b-bf28-f13cc9d99de4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-2p9rr" Nov 02 17:37:22.473723 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:22.473691 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz9w5\" (UniqueName: \"kubernetes.io/projected/85011b6f-dbf8-472b-bf28-f13cc9d99de4-kube-api-access-hz9w5\") pod \"payload-pre-processing-7b98bb678c-2p9rr\" (UID: \"85011b6f-dbf8-472b-bf28-f13cc9d99de4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-2p9rr" Nov 02 17:37:22.549134 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:22.549098 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-446pp\" (UniqueName: \"kubernetes.io/projected/91ff9502-0ff4-43a1-8f78-b5d8c8fe683e-kube-api-access-446pp\") pod \"payload-processing-55fbbbf9dc-rjwjd\" (UID: \"91ff9502-0ff4-43a1-8f78-b5d8c8fe683e\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rjwjd" Nov 02 17:37:22.549360 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:22.549147 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91ff9502-0ff4-43a1-8f78-b5d8c8fe683e-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-rjwjd\" (UID: \"91ff9502-0ff4-43a1-8f78-b5d8c8fe683e\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rjwjd" Nov 02 17:37:22.549897 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:22.549870 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91ff9502-0ff4-43a1-8f78-b5d8c8fe683e-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-rjwjd\" (UID: \"91ff9502-0ff4-43a1-8f78-b5d8c8fe683e\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rjwjd" Nov 02 17:37:22.574965 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:22.574899 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-446pp\" (UniqueName: \"kubernetes.io/projected/91ff9502-0ff4-43a1-8f78-b5d8c8fe683e-kube-api-access-446pp\") pod \"payload-processing-55fbbbf9dc-rjwjd\" (UID: \"91ff9502-0ff4-43a1-8f78-b5d8c8fe683e\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rjwjd" Nov 02 17:37:22.636467 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:22.636437 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-2p9rr" Nov 02 17:37:22.738790 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:22.738758 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-rjwjd" Nov 02 17:37:22.930036 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:22.930009 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-2p9rr"] Nov 02 17:37:22.930201 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:37:22.930181 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85011b6f_dbf8_472b_bf28_f13cc9d99de4.slice/crio-4157f9c15a104443722f9b33341cff06fa02fe1e897fe93736143a74cff5dca2 WatchSource:0}: Error finding container 4157f9c15a104443722f9b33341cff06fa02fe1e897fe93736143a74cff5dca2: Status 404 returned error can't find the container with id 4157f9c15a104443722f9b33341cff06fa02fe1e897fe93736143a74cff5dca2 Nov 02 17:37:23.030455 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:37:23.030424 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ff9502_0ff4_43a1_8f78_b5d8c8fe683e.slice/crio-992147fdecba0f4365926ec6c8d64a18e070e0b2bfbd76ef7a83070fe7ae6841 WatchSource:0}: Error finding container 992147fdecba0f4365926ec6c8d64a18e070e0b2bfbd76ef7a83070fe7ae6841: Status 404 returned error can't find the container with id 992147fdecba0f4365926ec6c8d64a18e070e0b2bfbd76ef7a83070fe7ae6841 Nov 02 17:37:23.030671 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:23.030652 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-rjwjd"] Nov 02 17:37:23.296994 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:23.296959 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-rjwjd" event={"ID":"91ff9502-0ff4-43a1-8f78-b5d8c8fe683e","Type":"ContainerStarted","Data":"992147fdecba0f4365926ec6c8d64a18e070e0b2bfbd76ef7a83070fe7ae6841"} Nov 02 17:37:23.298356 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:23.298330 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-2p9rr" event={"ID":"85011b6f-dbf8-472b-bf28-f13cc9d99de4","Type":"ContainerStarted","Data":"4157f9c15a104443722f9b33341cff06fa02fe1e897fe93736143a74cff5dca2"} Nov 02 17:37:26.314435 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:26.314395 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-2p9rr" event={"ID":"85011b6f-dbf8-472b-bf28-f13cc9d99de4","Type":"ContainerStarted","Data":"e612a831d89e8b1388f2d579311d6e21f130cd0945ce9cd8ba6987652f68249d"} Nov 02 17:37:26.314903 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:26.314470 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-2p9rr" Nov 02 17:37:26.315943 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:26.315921 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-rjwjd" event={"ID":"91ff9502-0ff4-43a1-8f78-b5d8c8fe683e","Type":"ContainerStarted","Data":"25891b9190932971f615836cdc0eccbfd55f2ba965c69608ce78164aa4f478c5"} Nov 02 17:37:26.316069 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:26.316058 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-rjwjd" Nov 02 17:37:26.356969 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:26.356923 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-2p9rr" podStartSLOduration=2.002514811 podStartE2EDuration="4.356909195s" totalImagesPullingTime="2.354394384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:37:26.356238705 +0000 UTC m=+464.051594175" watchObservedRunningTime="2026-11-02 17:37:26.356909195 +0000 UTC m=+464.052264645" Nov 02 17:37:26.398944 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:26.398899 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-rjwjd" podStartSLOduration=2.084988503 podStartE2EDuration="4.398884815s" totalImagesPullingTime="2.313896312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:37:26.397679658 +0000 UTC m=+464.093035131" watchObservedRunningTime="2026-11-02 17:37:26.398884815 +0000 UTC m=+464.094240287" Nov 02 17:37:31.966516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:31.966483 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5677bb6689-t5vq2"] Nov 02 17:37:31.970549 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:31.970529 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5677bb6689-t5vq2" Nov 02 17:37:31.999062 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:31.999038 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 02 17:37:31.999337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:31.999321 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 02 17:37:31.999517 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:31.999498 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-rwt92\"" Nov 02 17:37:32.000614 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.000595 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5677bb6689-t5vq2"] Nov 02 17:37:32.032688 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.032655 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8666c0bd-1a97-4bb3-8695-8f84b7047250-metrics-certs\") pod \"ai-gateway-controller-5677bb6689-t5vq2\" (UID: \"8666c0bd-1a97-4bb3-8695-8f84b7047250\") " pod="opendatahub/ai-gateway-controller-5677bb6689-t5vq2" Nov 02 17:37:32.032836 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.032689 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svmkw\" (UniqueName: \"kubernetes.io/projected/8666c0bd-1a97-4bb3-8695-8f84b7047250-kube-api-access-svmkw\") pod \"ai-gateway-controller-5677bb6689-t5vq2\" (UID: \"8666c0bd-1a97-4bb3-8695-8f84b7047250\") " pod="opendatahub/ai-gateway-controller-5677bb6689-t5vq2" Nov 02 17:37:32.133232 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.133206 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8666c0bd-1a97-4bb3-8695-8f84b7047250-metrics-certs\") pod \"ai-gateway-controller-5677bb6689-t5vq2\" (UID: \"8666c0bd-1a97-4bb3-8695-8f84b7047250\") " pod="opendatahub/ai-gateway-controller-5677bb6689-t5vq2" Nov 02 17:37:32.133365 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.133238 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svmkw\" (UniqueName: \"kubernetes.io/projected/8666c0bd-1a97-4bb3-8695-8f84b7047250-kube-api-access-svmkw\") pod \"ai-gateway-controller-5677bb6689-t5vq2\" (UID: \"8666c0bd-1a97-4bb3-8695-8f84b7047250\") " pod="opendatahub/ai-gateway-controller-5677bb6689-t5vq2" Nov 02 17:37:32.135578 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.135556 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8666c0bd-1a97-4bb3-8695-8f84b7047250-metrics-certs\") pod \"ai-gateway-controller-5677bb6689-t5vq2\" (UID: \"8666c0bd-1a97-4bb3-8695-8f84b7047250\") " pod="opendatahub/ai-gateway-controller-5677bb6689-t5vq2" Nov 02 17:37:32.157555 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.157529 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svmkw\" (UniqueName: \"kubernetes.io/projected/8666c0bd-1a97-4bb3-8695-8f84b7047250-kube-api-access-svmkw\") pod \"ai-gateway-controller-5677bb6689-t5vq2\" (UID: \"8666c0bd-1a97-4bb3-8695-8f84b7047250\") " pod="opendatahub/ai-gateway-controller-5677bb6689-t5vq2" Nov 02 17:37:32.281619 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.281595 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5677bb6689-t5vq2" Nov 02 17:37:32.582922 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.582896 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5677bb6689-t5vq2"] Nov 02 17:37:32.586783 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:37:32.586752 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8666c0bd_1a97_4bb3_8695_8f84b7047250.slice/crio-298fcd45345dc16f6266115c87e2f9ae5b1c5fffdc92a040fba20db9d52b2062 WatchSource:0}: Error finding container 298fcd45345dc16f6266115c87e2f9ae5b1c5fffdc92a040fba20db9d52b2062: Status 404 returned error can't find the container with id 298fcd45345dc16f6266115c87e2f9ae5b1c5fffdc92a040fba20db9d52b2062 Nov 02 17:37:32.802225 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.802193 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5"] Nov 02 17:37:32.805277 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.805256 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:37:32.834579 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.834516 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5"] Nov 02 17:37:32.840356 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.840300 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-8mnq5\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:37:32.840483 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.840435 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7689ac62-6665-4a39-80f2-1a3785898881-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-8mnq5\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:37:32.840533 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.840485 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7689ac62-6665-4a39-80f2-1a3785898881-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-8mnq5\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:37:32.840592 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.840574 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-8mnq5\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:37:32.840669 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.840622 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7689ac62-6665-4a39-80f2-1a3785898881-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-8mnq5\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:37:32.840669 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.840651 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-8mnq5\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:37:32.840846 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.840804 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-8mnq5\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:37:32.840916 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.840885 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-8mnq5\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:37:32.840978 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.840929 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7jqr\" (UniqueName: \"kubernetes.io/projected/7689ac62-6665-4a39-80f2-1a3785898881-kube-api-access-f7jqr\") pod \"maas-default-gateway-openshift-default-76f4876667-8mnq5\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:37:32.900639 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.900611 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-lkf8q\"" Nov 02 17:37:32.942072 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.942040 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7689ac62-6665-4a39-80f2-1a3785898881-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-8mnq5\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:37:32.942258 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.942087 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-8mnq5\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:37:32.942258 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.942221 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7689ac62-6665-4a39-80f2-1a3785898881-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-8mnq5\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:37:32.942416 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.942257 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-8mnq5\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:37:32.942475 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.942452 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-8mnq5\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:37:32.942546 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.942527 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-8mnq5\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:37:32.942601 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.942567 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7jqr\" (UniqueName: \"kubernetes.io/projected/7689ac62-6665-4a39-80f2-1a3785898881-kube-api-access-f7jqr\") pod \"maas-default-gateway-openshift-default-76f4876667-8mnq5\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:37:32.942663 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.942605 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-8mnq5\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:37:32.942663 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.942604 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-8mnq5\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:37:32.942769 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.942735 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7689ac62-6665-4a39-80f2-1a3785898881-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-8mnq5\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:37:32.942925 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.942875 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-8mnq5\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:37:32.943387 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.943351 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-8mnq5\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:37:32.943491 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.943477 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7689ac62-6665-4a39-80f2-1a3785898881-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-8mnq5\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:37:32.943566 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.943503 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-8mnq5\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:37:32.949115 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.944821 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-8mnq5\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:37:32.949115 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.945220 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7689ac62-6665-4a39-80f2-1a3785898881-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-8mnq5\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:37:32.979410 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.979377 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7jqr\" (UniqueName: \"kubernetes.io/projected/7689ac62-6665-4a39-80f2-1a3785898881-kube-api-access-f7jqr\") pod \"maas-default-gateway-openshift-default-76f4876667-8mnq5\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:37:32.979911 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:32.979889 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7689ac62-6665-4a39-80f2-1a3785898881-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-8mnq5\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:37:33.119748 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:33.119672 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:37:33.350324 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:33.350262 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5677bb6689-t5vq2" event={"ID":"8666c0bd-1a97-4bb3-8695-8f84b7047250","Type":"ContainerStarted","Data":"298fcd45345dc16f6266115c87e2f9ae5b1c5fffdc92a040fba20db9d52b2062"} Nov 02 17:37:33.432668 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:33.432641 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5"] Nov 02 17:37:34.356628 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:34.356586 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" event={"ID":"7689ac62-6665-4a39-80f2-1a3785898881","Type":"ContainerStarted","Data":"6257aa7cab76b8858088eca1413f0d2f3e039c0a3a8897c9c3058b3bc5610e60"} Nov 02 17:37:34.358335 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:34.358284 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5677bb6689-t5vq2" event={"ID":"8666c0bd-1a97-4bb3-8695-8f84b7047250","Type":"ContainerStarted","Data":"94ed6364e2173dd730ef3df5626b0465b4ab1c2a7f18cd69cca6721200329997"} Nov 02 17:37:34.358487 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:34.358436 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5677bb6689-t5vq2" Nov 02 17:37:34.401427 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:34.401362 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5677bb6689-t5vq2" podStartSLOduration=2.022445057 podStartE2EDuration="3.401341864s" totalImagesPullingTime="1.378896807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:37:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:37:34.400855152 +0000 UTC m=+472.096210637" watchObservedRunningTime="2026-11-02 17:37:34.401341864 +0000 UTC m=+472.096697337" Nov 02 17:37:36.298115 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:36.298085 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:37:36.298403 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:36.298151 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:37:36.298403 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:36.298185 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:37:36.368381 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:36.368350 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" event={"ID":"7689ac62-6665-4a39-80f2-1a3785898881","Type":"ContainerStarted","Data":"c556e00298ef95e9632c853cda6e55e2bf0bc1f340f88982b09be25ef1441379"} Nov 02 17:37:36.421721 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:36.421660 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" podStartSLOduration=1.5625642339999999 podStartE2EDuration="4.42163862s" totalImagesPullingTime="2.859074386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:37:36.42052674 +0000 UTC m=+474.115882240" watchObservedRunningTime="2026-11-02 17:37:36.42163862 +0000 UTC m=+474.116994095" Nov 02 17:37:37.120754 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:37.120717 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:37:37.122233 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:37.122202 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-8mnq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 02 17:37:37.122357 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:37.122251 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" podUID="7689ac62-6665-4a39-80f2-1a3785898881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 02 17:37:37.323704 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:37.323670 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-2p9rr" Nov 02 17:37:37.324111 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:37.323737 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-rjwjd" Nov 02 17:37:38.120457 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:38.120419 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-8mnq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 02 17:37:38.120667 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:38.120477 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" podUID="7689ac62-6665-4a39-80f2-1a3785898881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 02 17:37:39.120762 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:39.120727 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-8mnq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 02 17:37:39.121155 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:39.120797 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" podUID="7689ac62-6665-4a39-80f2-1a3785898881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 02 17:37:39.205495 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:37:39.205461 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:37:40.120024 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:40.119987 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-8mnq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 02 17:37:40.120201 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:40.120040 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" podUID="7689ac62-6665-4a39-80f2-1a3785898881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 02 17:37:41.120507 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:41.120474 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-8mnq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 02 17:37:41.120905 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:41.120535 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" podUID="7689ac62-6665-4a39-80f2-1a3785898881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 02 17:37:42.120960 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:42.120925 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-8mnq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 02 17:37:42.121379 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:42.120976 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" podUID="7689ac62-6665-4a39-80f2-1a3785898881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 02 17:37:43.120205 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:43.120161 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-8mnq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 02 17:37:43.120412 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:43.120237 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" podUID="7689ac62-6665-4a39-80f2-1a3785898881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 02 17:37:44.549948 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:44.549914 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:37:44.550450 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:44.550344 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:37:44.551164 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:44.551144 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:37:45.365190 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:45.365160 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5677bb6689-t5vq2" Nov 02 17:37:47.824275 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.824235 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl"] Nov 02 17:37:47.831470 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.831331 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:37:47.857956 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.857919 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc"] Nov 02 17:37:47.861897 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.861872 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 17:37:47.862824 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.862794 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl"] Nov 02 17:37:47.888437 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.888403 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-5zqhl\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:37:47.888437 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.888437 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-5zqhl\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:37:47.888638 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.888614 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5zqhl\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:37:47.888704 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.888685 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-5zqhl\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:37:47.888753 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.888725 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-5zqhl\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:37:47.888806 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.888777 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5zqhl\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:37:47.888899 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.888878 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-5zqhl\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:37:47.888973 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.888927 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-5zqhl\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:37:47.888973 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.888957 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwqdq\" (UniqueName: \"kubernetes.io/projected/398b95b8-a6ae-442f-ba56-d9c77e4b2050-kube-api-access-vwqdq\") pod \"maas-default-gateway-openshift-default-76f4876667-5zqhl\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:37:47.893188 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.893164 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc"] Nov 02 17:37:47.989706 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.989675 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwqdq\" (UniqueName: \"kubernetes.io/projected/398b95b8-a6ae-442f-ba56-d9c77e4b2050-kube-api-access-vwqdq\") pod \"maas-default-gateway-openshift-default-76f4876667-5zqhl\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:37:47.989706 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.989714 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-5zqhl\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:37:47.989965 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.989739 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-n75gc\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 17:37:47.989965 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.989765 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-n75gc\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 17:37:47.989965 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.989850 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-n75gc\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 17:37:47.989965 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.989882 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-5zqhl\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:37:47.989965 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.989929 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-n75gc\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 17:37:47.989965 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.989952 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-n75gc\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 17:37:47.990287 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.990025 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-n75gc\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 17:37:47.990287 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.990065 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-5zqhl\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:37:47.990287 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.990091 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-5zqhl\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:37:47.990287 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.990166 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-n75gc\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 17:37:47.990287 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.990177 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-5zqhl\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:37:47.990287 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.990200 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5zqhl\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:37:47.990287 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.990242 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szssp\" (UniqueName: \"kubernetes.io/projected/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-kube-api-access-szssp\") pod \"maas-default-gateway-openshift-default-76f4876667-n75gc\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 17:37:47.990684 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.990368 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-5zqhl\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:37:47.990684 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.990399 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-n75gc\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 17:37:47.990684 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.990432 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5zqhl\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:37:47.990684 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.990473 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-5zqhl\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:37:47.990684 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.990491 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5zqhl\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:37:47.990884 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.990697 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-5zqhl\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:37:47.990884 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.990777 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5zqhl\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:37:47.990884 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.990852 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-5zqhl\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:37:47.992639 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.992618 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-5zqhl\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:37:47.992961 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:47.992941 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-5zqhl\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:37:48.013793 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:48.013770 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-5zqhl\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:37:48.013896 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:48.013858 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwqdq\" (UniqueName: \"kubernetes.io/projected/398b95b8-a6ae-442f-ba56-d9c77e4b2050-kube-api-access-vwqdq\") pod \"maas-default-gateway-openshift-default-76f4876667-5zqhl\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:37:48.091762 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:48.091700 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-n75gc\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 17:37:48.091762 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:48.091734 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szssp\" (UniqueName: \"kubernetes.io/projected/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-kube-api-access-szssp\") pod \"maas-default-gateway-openshift-default-76f4876667-n75gc\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 17:37:48.091933 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:48.091769 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-n75gc\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 17:37:48.091933 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:48.091850 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-n75gc\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 17:37:48.091933 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:48.091902 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-n75gc\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 17:37:48.092063 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:48.091933 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-n75gc\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 17:37:48.092063 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:48.091962 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-n75gc\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 17:37:48.092063 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:48.091994 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-n75gc\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 17:37:48.092063 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:48.092021 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-n75gc\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 17:37:48.092238 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:48.092127 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-n75gc\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 17:37:48.092238 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:48.092217 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-n75gc\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 17:37:48.092379 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:48.092345 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-n75gc\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 17:37:48.092460 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:48.092443 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-n75gc\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 17:37:48.092683 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:48.092661 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-n75gc\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 17:37:48.094108 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:48.094088 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-n75gc\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 17:37:48.094492 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:48.094474 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-n75gc\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 17:37:48.115720 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:48.115697 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-n75gc\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 17:37:48.116020 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:48.116003 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szssp\" (UniqueName: \"kubernetes.io/projected/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-kube-api-access-szssp\") pod \"maas-default-gateway-openshift-default-76f4876667-n75gc\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 17:37:48.149976 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:48.149951 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:37:48.177336 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:48.177297 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 17:37:48.462907 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:48.462883 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl"] Nov 02 17:37:48.463435 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:37:48.463405 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod398b95b8_a6ae_442f_ba56_d9c77e4b2050.slice/crio-ee45f63b60ef9a23581c7c646c87d9c0c412cf9ebadac6c79faf9970fd61475c WatchSource:0}: Error finding container ee45f63b60ef9a23581c7c646c87d9c0c412cf9ebadac6c79faf9970fd61475c: Status 404 returned error can't find the container with id ee45f63b60ef9a23581c7c646c87d9c0c412cf9ebadac6c79faf9970fd61475c Nov 02 17:37:48.465578 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:48.465550 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:37:48.465658 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:48.465611 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:37:48.465658 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:48.465643 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:37:48.499448 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:48.499425 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc"] Nov 02 17:37:48.502389 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:37:48.502366 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ebaee6a_d3f4_4f0d_8485_6dd565c1e078.slice/crio-62120c3ffe86bce5e25b609d63c46bcf43f032a9f95030dec194f2125b583ad8 WatchSource:0}: Error finding container 62120c3ffe86bce5e25b609d63c46bcf43f032a9f95030dec194f2125b583ad8: Status 404 returned error can't find the container with id 62120c3ffe86bce5e25b609d63c46bcf43f032a9f95030dec194f2125b583ad8 Nov 02 17:37:48.504441 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:48.504411 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:37:48.504514 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:48.504477 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:37:48.504514 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:48.504503 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:37:49.446067 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:49.446025 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" event={"ID":"398b95b8-a6ae-442f-ba56-d9c77e4b2050","Type":"ContainerStarted","Data":"501e013e74fce830a2279f8b4a8bb07af4b2a6a70172d11e33fd1eb5872157a5"} Nov 02 17:37:49.446563 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:49.446079 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" event={"ID":"398b95b8-a6ae-442f-ba56-d9c77e4b2050","Type":"ContainerStarted","Data":"ee45f63b60ef9a23581c7c646c87d9c0c412cf9ebadac6c79faf9970fd61475c"} Nov 02 17:37:49.447827 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:49.447802 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" event={"ID":"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078","Type":"ContainerStarted","Data":"3bb3b7e01e995024c94215d5661cd6350bed5f90ed1c81bb2169c1a1544fdc66"} Nov 02 17:37:49.447964 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:49.447846 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" event={"ID":"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078","Type":"ContainerStarted","Data":"62120c3ffe86bce5e25b609d63c46bcf43f032a9f95030dec194f2125b583ad8"} Nov 02 17:37:49.577329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:49.572972 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" podStartSLOduration=2.572949567 podStartE2EDuration="2.572949567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:37:49.515997156 +0000 UTC m=+487.211352629" watchObservedRunningTime="2026-11-02 17:37:49.572949567 +0000 UTC m=+487.268305041" Nov 02 17:37:50.151072 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:50.151031 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:37:50.152530 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:50.152501 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5zqhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 02 17:37:50.152657 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:50.152550 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" podUID="398b95b8-a6ae-442f-ba56-d9c77e4b2050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 02 17:37:50.180047 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:50.180013 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 17:37:50.180406 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:50.180382 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-n75gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 17:37:50.180604 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:50.180582 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" podUID="9ebaee6a-d3f4-4f0d-8485-6dd565c1e078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 17:37:51.151339 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:51.151271 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5zqhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 02 17:37:51.151723 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:51.151363 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" podUID="398b95b8-a6ae-442f-ba56-d9c77e4b2050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 02 17:37:51.178564 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:51.178511 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-n75gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 17:37:51.178883 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:51.178825 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" podUID="9ebaee6a-d3f4-4f0d-8485-6dd565c1e078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 17:37:52.151229 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:52.151192 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5zqhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 02 17:37:52.151453 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:52.151246 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" podUID="398b95b8-a6ae-442f-ba56-d9c77e4b2050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 02 17:37:52.181719 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:52.181684 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-n75gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 17:37:52.181967 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:52.181940 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" podUID="9ebaee6a-d3f4-4f0d-8485-6dd565c1e078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 17:37:53.151003 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:53.150967 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5zqhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 02 17:37:53.151187 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:53.151035 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" podUID="398b95b8-a6ae-442f-ba56-d9c77e4b2050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 02 17:37:53.189330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:53.188703 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-n75gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 17:37:53.189330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:53.188769 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" podUID="9ebaee6a-d3f4-4f0d-8485-6dd565c1e078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 17:37:54.150447 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:54.150414 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5zqhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 02 17:37:54.150699 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:54.150468 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" podUID="398b95b8-a6ae-442f-ba56-d9c77e4b2050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 02 17:37:54.181330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:54.180686 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-n75gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 17:37:54.181330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:54.180757 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" podUID="9ebaee6a-d3f4-4f0d-8485-6dd565c1e078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 17:37:55.151340 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:55.151238 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5zqhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 02 17:37:55.151720 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:55.151330 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" podUID="398b95b8-a6ae-442f-ba56-d9c77e4b2050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 02 17:37:55.177956 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:55.177921 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-n75gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 17:37:55.178093 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:55.177983 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" podUID="9ebaee6a-d3f4-4f0d-8485-6dd565c1e078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 17:37:56.783485 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:56.783453 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:37:56.783954 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:56.783851 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:37:56.784713 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:56.784635 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:37:56.807567 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:56.807544 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 17:37:56.807918 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:56.807869 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 17:37:56.808975 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:56.808955 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 17:37:56.846990 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:37:56.846945 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" podStartSLOduration=9.846932516 podStartE2EDuration="9.846932516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:37:49.574695681 +0000 UTC m=+487.270051174" watchObservedRunningTime="2026-11-02 17:37:56.846932516 +0000 UTC m=+494.542287993" Nov 02 17:38:09.181384 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:38:09.181342 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:38:15.389423 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.389387 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f"] Nov 02 17:38:15.395444 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.395411 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:15.412355 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.412326 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-5eedab-openshift-default-dockercfg-57c6m\"" Nov 02 17:38:15.428374 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.428319 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f"] Nov 02 17:38:15.481661 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.481625 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77sxd\" (UniqueName: \"kubernetes.io/projected/14703ce4-5dc1-43d0-8368-c217f0bab9eb-kube-api-access-77sxd\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:15.481661 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.481665 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-workload-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:15.481938 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.481684 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istiod-ca-cert\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:15.481938 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.481848 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istio-data\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:15.482034 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.481982 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-workload-certs\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:15.482034 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.482023 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istio-envoy\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:15.482111 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.482038 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istio-token\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:15.482147 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.482128 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istio-podinfo\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:15.482181 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.482154 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-credential-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:15.573964 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.573929 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l"] Nov 02 17:38:15.578409 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.578378 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:15.582775 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.582743 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49caa894-b067-4d85-8141-741e72c8f041-istio-podinfo\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:15.582922 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.582901 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-workload-certs\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:15.582993 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.582937 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvk57\" (UniqueName: \"kubernetes.io/projected/49caa894-b067-4d85-8141-741e72c8f041-kube-api-access-vvk57\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:15.582993 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.582973 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77sxd\" (UniqueName: \"kubernetes.io/projected/14703ce4-5dc1-43d0-8368-c217f0bab9eb-kube-api-access-77sxd\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:15.583094 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.583028 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-workload-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:15.583094 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.583066 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-istio-envoy\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:15.583195 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.583104 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istiod-ca-cert\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:15.583195 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.583141 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istio-data\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:15.583358 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.583224 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-workload-certs\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:15.583358 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.583264 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49caa894-b067-4d85-8141-741e72c8f041-istio-token\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:15.584354 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.583481 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istio-envoy\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:15.584354 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.583517 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istio-token\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:15.584354 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.583551 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-workload-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:15.584354 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.583609 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istio-podinfo\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:15.584354 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.583634 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-istio-data\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:15.584354 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.583652 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-workload-certs\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:15.584354 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.583678 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-credential-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:15.584354 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.583859 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istiod-ca-cert\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:15.584354 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.584012 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-credential-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:15.584354 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.584155 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-credential-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:15.584354 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.584259 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49caa894-b067-4d85-8141-741e72c8f041-istiod-ca-cert\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:15.585108 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.584378 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-workload-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:15.585603 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.585577 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istio-data\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:15.586542 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.586447 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istio-envoy\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:15.588041 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.588017 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istio-podinfo\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:15.605117 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.605094 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l"] Nov 02 17:38:15.612339 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.612289 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istio-token\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:15.612454 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.612413 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77sxd\" (UniqueName: \"kubernetes.io/projected/14703ce4-5dc1-43d0-8368-c217f0bab9eb-kube-api-access-77sxd\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:15.638451 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.638428 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-863b87-openshift-default-dockercfg-5mwm4\"" Nov 02 17:38:15.684778 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.684706 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-credential-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:15.684778 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.684748 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49caa894-b067-4d85-8141-741e72c8f041-istiod-ca-cert\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:15.684778 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.684768 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49caa894-b067-4d85-8141-741e72c8f041-istio-podinfo\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:15.685014 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.684798 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-workload-certs\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:15.685014 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.684821 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvk57\" (UniqueName: \"kubernetes.io/projected/49caa894-b067-4d85-8141-741e72c8f041-kube-api-access-vvk57\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:15.685014 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.684851 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-istio-envoy\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:15.685014 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.684915 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49caa894-b067-4d85-8141-741e72c8f041-istio-token\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:15.685014 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.684942 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-workload-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:15.685014 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.684980 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-istio-data\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:15.685322 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.685158 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-credential-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:15.685322 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.685171 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-workload-certs\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:15.685322 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.685244 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-istio-data\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:15.685543 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.685516 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-workload-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:15.685812 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.685794 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49caa894-b067-4d85-8141-741e72c8f041-istiod-ca-cert\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:15.687236 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.687211 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-istio-envoy\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:15.687351 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.687214 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49caa894-b067-4d85-8141-741e72c8f041-istio-podinfo\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:15.708263 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.708238 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:15.713240 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.713217 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvk57\" (UniqueName: \"kubernetes.io/projected/49caa894-b067-4d85-8141-741e72c8f041-kube-api-access-vvk57\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:15.713560 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.713544 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49caa894-b067-4d85-8141-741e72c8f041-istio-token\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:15.908755 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:15.908725 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:16.035753 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:38:16.035720 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14703ce4_5dc1_43d0_8368_c217f0bab9eb.slice/crio-c02cc3f9d8a639396d773139a1106693b595895d1155fae053944573f2383493 WatchSource:0}: Error finding container c02cc3f9d8a639396d773139a1106693b595895d1155fae053944573f2383493: Status 404 returned error can't find the container with id c02cc3f9d8a639396d773139a1106693b595895d1155fae053944573f2383493 Nov 02 17:38:16.035879 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.035847 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f"] Nov 02 17:38:16.037995 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.037966 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:38:16.038064 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.038028 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:38:16.038064 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.038060 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:38:16.078183 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.078136 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4"] Nov 02 17:38:16.082263 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.082241 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:16.089587 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.089065 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006cd169-c635-4f97-a8d1-c21aa6c077c3-istiod-ca-cert\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:16.089587 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.089130 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-workload-certs\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:16.089587 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.089170 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-credential-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:16.089587 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.089214 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006cd169-c635-4f97-a8d1-c21aa6c077c3-istio-podinfo\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:16.089587 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.089248 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wc88\" (UniqueName: \"kubernetes.io/projected/006cd169-c635-4f97-a8d1-c21aa6c077c3-kube-api-access-8wc88\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:16.089587 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.089278 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006cd169-c635-4f97-a8d1-c21aa6c077c3-istio-token\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:16.089587 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.089325 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-istio-envoy\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:16.089587 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.089354 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-istio-data\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:16.089587 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.089424 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-workload-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:16.106897 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.106866 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-2899ab-openshift-default-dockercfg-t6vss\"" Nov 02 17:38:16.107182 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.107157 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4"] Nov 02 17:38:16.193073 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.193026 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006cd169-c635-4f97-a8d1-c21aa6c077c3-istiod-ca-cert\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:16.216384 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.210423 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-workload-certs\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:16.216384 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.210502 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-credential-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:16.216384 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.210559 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006cd169-c635-4f97-a8d1-c21aa6c077c3-istio-podinfo\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:16.216384 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.210599 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wc88\" (UniqueName: \"kubernetes.io/projected/006cd169-c635-4f97-a8d1-c21aa6c077c3-kube-api-access-8wc88\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:16.216384 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.210637 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006cd169-c635-4f97-a8d1-c21aa6c077c3-istio-token\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:16.216384 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.210709 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-istio-envoy\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:16.216384 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.210746 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-istio-data\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:16.216384 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.210814 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-workload-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:16.216384 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.211791 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-workload-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:16.216384 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.212058 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-istio-data\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:16.216384 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.212280 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-workload-certs\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:16.216384 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.194855 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006cd169-c635-4f97-a8d1-c21aa6c077c3-istiod-ca-cert\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:16.216384 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.212586 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-credential-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:16.228464 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.228382 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l"] Nov 02 17:38:16.257347 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.240640 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-istio-envoy\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:16.257347 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.241546 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006cd169-c635-4f97-a8d1-c21aa6c077c3-istio-podinfo\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:16.257347 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.244362 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006cd169-c635-4f97-a8d1-c21aa6c077c3-istio-token\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:16.257347 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.244550 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wc88\" (UniqueName: \"kubernetes.io/projected/006cd169-c635-4f97-a8d1-c21aa6c077c3-kube-api-access-8wc88\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:16.269537 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.269477 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:38:16.269756 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.269733 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:38:16.269904 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.269884 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:38:16.398816 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.398069 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:16.620784 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.620747 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" event={"ID":"49caa894-b067-4d85-8141-741e72c8f041","Type":"ContainerStarted","Data":"1d452628e1a1a514cdc498bf3e85994af2e768ec805be2df240146fedfe81cbe"} Nov 02 17:38:16.620984 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.620810 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" event={"ID":"49caa894-b067-4d85-8141-741e72c8f041","Type":"ContainerStarted","Data":"9b6fc0af8f78729ba08112f5f23719b1d73166959077a306a87cca2b1969547b"} Nov 02 17:38:16.622590 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.622563 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" event={"ID":"14703ce4-5dc1-43d0-8368-c217f0bab9eb","Type":"ContainerStarted","Data":"1972f28c65a6080b0770ac769426a03f8ef87c57fca1079e1a53948cfd1550de"} Nov 02 17:38:16.622687 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.622608 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" event={"ID":"14703ce4-5dc1-43d0-8368-c217f0bab9eb","Type":"ContainerStarted","Data":"c02cc3f9d8a639396d773139a1106693b595895d1155fae053944573f2383493"} Nov 02 17:38:16.708822 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.708788 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:16.720096 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.720044 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" podStartSLOduration=1.720029225 podStartE2EDuration="1.720029225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:16.717679917 +0000 UTC m=+514.413035400" watchObservedRunningTime="2026-11-02 17:38:16.720029225 +0000 UTC m=+514.415384697" Nov 02 17:38:16.766372 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:38:16.764288 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod006cd169_c635_4f97_a8d1_c21aa6c077c3.slice/crio-0d05c7d1678f808d68e63e396312e0f10bf557f955bc080013e8ba019e8a5247 WatchSource:0}: Error finding container 0d05c7d1678f808d68e63e396312e0f10bf557f955bc080013e8ba019e8a5247: Status 404 returned error can't find the container with id 0d05c7d1678f808d68e63e396312e0f10bf557f955bc080013e8ba019e8a5247 Nov 02 17:38:16.766690 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.766633 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4"] Nov 02 17:38:16.767483 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.767455 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:38:16.767533 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.767514 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:38:16.767577 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.767544 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:38:16.793241 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.793195 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" podStartSLOduration=1.793180628 podStartE2EDuration="1.793180628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:16.791823124 +0000 UTC m=+514.487178598" watchObservedRunningTime="2026-11-02 17:38:16.793180628 +0000 UTC m=+514.488536100" Nov 02 17:38:16.911821 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:16.911725 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:17.628855 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:17.628817 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" event={"ID":"006cd169-c635-4f97-a8d1-c21aa6c077c3","Type":"ContainerStarted","Data":"366bef4b47ff7c4e99b8ebe59103d9643b3e031789c9362d72ef563b40aff5c5"} Nov 02 17:38:17.629241 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:17.628876 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" event={"ID":"006cd169-c635-4f97-a8d1-c21aa6c077c3","Type":"ContainerStarted","Data":"0d05c7d1678f808d68e63e396312e0f10bf557f955bc080013e8ba019e8a5247"} Nov 02 17:38:17.705892 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:17.705828 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" podStartSLOduration=1.705813324 podStartE2EDuration="1.705813324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:17.693211251 +0000 UTC m=+515.388566765" watchObservedRunningTime="2026-11-02 17:38:17.705813324 +0000 UTC m=+515.401168799" Nov 02 17:38:17.713734 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:17.713706 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:17.915354 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:17.915256 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:18.399430 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:18.399393 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:18.404899 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:18.404874 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:18.633328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:18.633264 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:18.633804 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:18.633600 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:18.633804 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:18.633645 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:18.634297 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:18.634274 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:18.634459 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:18.634437 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:18.634526 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:18.634496 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:22.654053 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:22.654019 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-120394-854c949567-5xfqn"] Nov 02 17:38:22.657313 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:22.657288 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-120394-854c949567-5xfqn" Nov 02 17:38:22.688194 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:22.688162 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-120394-serving-cert\"" Nov 02 17:38:22.688654 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:22.688626 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9-maas-api-tls\") pod \"maas-api-e2e-worker-w1-120394-854c949567-5xfqn\" (UID: \"7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-120394-854c949567-5xfqn" Nov 02 17:38:22.688772 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:22.688747 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-120394-854c949567-5xfqn\" (UID: \"7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-120394-854c949567-5xfqn" Nov 02 17:38:22.688886 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:22.688871 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzk44\" (UniqueName: \"kubernetes.io/projected/7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9-kube-api-access-pzk44\") pod \"maas-api-e2e-worker-w1-120394-854c949567-5xfqn\" (UID: \"7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-120394-854c949567-5xfqn" Nov 02 17:38:22.688953 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:22.688936 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9-metrics-certs\") pod \"maas-api-e2e-worker-w1-120394-854c949567-5xfqn\" (UID: \"7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-120394-854c949567-5xfqn" Nov 02 17:38:22.692126 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:22.692100 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-120394-854c949567-5xfqn"] Nov 02 17:38:22.790317 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:22.790281 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9-maas-api-tls\") pod \"maas-api-e2e-worker-w1-120394-854c949567-5xfqn\" (UID: \"7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-120394-854c949567-5xfqn" Nov 02 17:38:22.790492 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:22.790359 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-120394-854c949567-5xfqn\" (UID: \"7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-120394-854c949567-5xfqn" Nov 02 17:38:22.790492 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:38:22.790410 4471 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-120394-serving-cert: secret "maas-api-e2e-worker-w1-120394-serving-cert" not found Nov 02 17:38:22.790492 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:38:22.790479 4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9-maas-api-tls podName:7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9 nodeName:}" failed. No retries permitted until 2026-11-02 17:38:23.290461316 +0000 UTC m=+520.985816767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9-maas-api-tls") pod "maas-api-e2e-worker-w1-120394-854c949567-5xfqn" (UID: "7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9") : secret "maas-api-e2e-worker-w1-120394-serving-cert" not found Nov 02 17:38:22.790656 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:22.790413 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzk44\" (UniqueName: \"kubernetes.io/projected/7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9-kube-api-access-pzk44\") pod \"maas-api-e2e-worker-w1-120394-854c949567-5xfqn\" (UID: \"7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-120394-854c949567-5xfqn" Nov 02 17:38:22.790656 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:22.790578 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9-metrics-certs\") pod \"maas-api-e2e-worker-w1-120394-854c949567-5xfqn\" (UID: \"7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-120394-854c949567-5xfqn" Nov 02 17:38:22.791085 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:22.791056 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-120394-854c949567-5xfqn\" (UID: \"7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-120394-854c949567-5xfqn" Nov 02 17:38:22.793091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:22.793070 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9-metrics-certs\") pod \"maas-api-e2e-worker-w1-120394-854c949567-5xfqn\" (UID: \"7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-120394-854c949567-5xfqn" Nov 02 17:38:22.830932 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:22.830905 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzk44\" (UniqueName: \"kubernetes.io/projected/7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9-kube-api-access-pzk44\") pod \"maas-api-e2e-worker-w1-120394-854c949567-5xfqn\" (UID: \"7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-120394-854c949567-5xfqn" Nov 02 17:38:23.294896 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:23.294863 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9-maas-api-tls\") pod \"maas-api-e2e-worker-w1-120394-854c949567-5xfqn\" (UID: \"7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-120394-854c949567-5xfqn" Nov 02 17:38:23.297237 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:23.297205 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9-maas-api-tls\") pod \"maas-api-e2e-worker-w1-120394-854c949567-5xfqn\" (UID: \"7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-120394-854c949567-5xfqn" Nov 02 17:38:23.577915 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:23.577833 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-120394-854c949567-5xfqn" Nov 02 17:38:23.906705 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:23.906671 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-120394-854c949567-5xfqn"] Nov 02 17:38:23.907231 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:38:23.907201 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d7f0b28_b3a9_4943_b2ae_fa47d66cb4b9.slice/crio-c0902f4d39ce1089949658a8c2d37e78400e2c711f84e04309d0d7ce0f696d8c WatchSource:0}: Error finding container c0902f4d39ce1089949658a8c2d37e78400e2c711f84e04309d0d7ce0f696d8c: Status 404 returned error can't find the container with id c0902f4d39ce1089949658a8c2d37e78400e2c711f84e04309d0d7ce0f696d8c Nov 02 17:38:24.668445 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:24.668357 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-120394-854c949567-5xfqn" event={"ID":"7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9","Type":"ContainerStarted","Data":"e584069d5f2542331f41acfb8ff40e1669c901fb178f7d100ce666f2868873ff"} Nov 02 17:38:24.668445 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:24.668414 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-120394-854c949567-5xfqn" event={"ID":"7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9","Type":"ContainerStarted","Data":"c0902f4d39ce1089949658a8c2d37e78400e2c711f84e04309d0d7ce0f696d8c"} Nov 02 17:38:24.668616 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:24.668463 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-120394-854c949567-5xfqn" Nov 02 17:38:24.724635 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:24.724586 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-120394-854c949567-5xfqn" podStartSLOduration=2.232660595 podStartE2EDuration="2.724573063s" totalImagesPullingTime="491.912468ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:24.724420741 +0000 UTC m=+522.419776228" watchObservedRunningTime="2026-11-02 17:38:24.724573063 +0000 UTC m=+522.419928536" Nov 02 17:38:26.871091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:26.871038 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:38:26.966592 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:26.966560 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6"] Nov 02 17:38:26.970358 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:26.970339 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6" Nov 02 17:38:27.003422 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:27.003390 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6"] Nov 02 17:38:27.003594 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:27.003498 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-7df526-serving-cert\"" Nov 02 17:38:27.035439 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:27.035401 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d5d1db2-0240-442b-aaed-b2931e9e8d6a-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6\" (UID: \"4d5d1db2-0240-442b-aaed-b2931e9e8d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6" Nov 02 17:38:27.035611 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:27.035591 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d5d1db2-0240-442b-aaed-b2931e9e8d6a-metrics-certs\") pod \"maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6\" (UID: \"4d5d1db2-0240-442b-aaed-b2931e9e8d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6" Nov 02 17:38:27.035684 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:27.035632 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d5d1db2-0240-442b-aaed-b2931e9e8d6a-maas-api-tls\") pod \"maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6\" (UID: \"4d5d1db2-0240-442b-aaed-b2931e9e8d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6" Nov 02 17:38:27.035745 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:27.035710 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbn5r\" (UniqueName: \"kubernetes.io/projected/4d5d1db2-0240-442b-aaed-b2931e9e8d6a-kube-api-access-fbn5r\") pod \"maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6\" (UID: \"4d5d1db2-0240-442b-aaed-b2931e9e8d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6" Nov 02 17:38:27.137001 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:27.136917 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d5d1db2-0240-442b-aaed-b2931e9e8d6a-maas-api-tls\") pod \"maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6\" (UID: \"4d5d1db2-0240-442b-aaed-b2931e9e8d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6" Nov 02 17:38:27.137001 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:27.136952 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbn5r\" (UniqueName: \"kubernetes.io/projected/4d5d1db2-0240-442b-aaed-b2931e9e8d6a-kube-api-access-fbn5r\") pod \"maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6\" (UID: \"4d5d1db2-0240-442b-aaed-b2931e9e8d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6" Nov 02 17:38:27.137001 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:27.136989 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d5d1db2-0240-442b-aaed-b2931e9e8d6a-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6\" (UID: \"4d5d1db2-0240-442b-aaed-b2931e9e8d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6" Nov 02 17:38:27.137287 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:27.137107 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d5d1db2-0240-442b-aaed-b2931e9e8d6a-metrics-certs\") pod \"maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6\" (UID: \"4d5d1db2-0240-442b-aaed-b2931e9e8d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6" Nov 02 17:38:27.137700 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:27.137675 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d5d1db2-0240-442b-aaed-b2931e9e8d6a-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6\" (UID: \"4d5d1db2-0240-442b-aaed-b2931e9e8d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6" Nov 02 17:38:27.139457 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:27.139434 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d5d1db2-0240-442b-aaed-b2931e9e8d6a-metrics-certs\") pod \"maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6\" (UID: \"4d5d1db2-0240-442b-aaed-b2931e9e8d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6" Nov 02 17:38:27.139557 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:27.139492 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d5d1db2-0240-442b-aaed-b2931e9e8d6a-maas-api-tls\") pod \"maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6\" (UID: \"4d5d1db2-0240-442b-aaed-b2931e9e8d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6" Nov 02 17:38:27.162808 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:27.162779 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbn5r\" (UniqueName: \"kubernetes.io/projected/4d5d1db2-0240-442b-aaed-b2931e9e8d6a-kube-api-access-fbn5r\") pod \"maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6\" (UID: \"4d5d1db2-0240-442b-aaed-b2931e9e8d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6" Nov 02 17:38:27.283922 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:27.283887 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6" Nov 02 17:38:27.653188 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:27.653163 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6"] Nov 02 17:38:27.653350 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:38:27.653326 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d5d1db2_0240_442b_aaed_b2931e9e8d6a.slice/crio-bcb2bded9562805957ac41a7a94f18761dd9cf2a8505b0ebbe83a3f4a70065c0 WatchSource:0}: Error finding container bcb2bded9562805957ac41a7a94f18761dd9cf2a8505b0ebbe83a3f4a70065c0: Status 404 returned error can't find the container with id bcb2bded9562805957ac41a7a94f18761dd9cf2a8505b0ebbe83a3f4a70065c0 Nov 02 17:38:27.683452 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:27.683423 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6" event={"ID":"4d5d1db2-0240-442b-aaed-b2931e9e8d6a","Type":"ContainerStarted","Data":"bcb2bded9562805957ac41a7a94f18761dd9cf2a8505b0ebbe83a3f4a70065c0"} Nov 02 17:38:28.690177 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:28.690145 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6" event={"ID":"4d5d1db2-0240-442b-aaed-b2931e9e8d6a","Type":"ContainerStarted","Data":"f70838619a28f2c43aaf4898798aa5898183aa45ec70deeacfce753b3c981a3e"} Nov 02 17:38:28.690594 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:28.690289 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6" Nov 02 17:38:28.740495 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:28.740432 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6" podStartSLOduration=2.390521332 podStartE2EDuration="2.740416811s" totalImagesPullingTime="349.895479ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:28.739454446 +0000 UTC m=+526.434809921" watchObservedRunningTime="2026-11-02 17:38:28.740416811 +0000 UTC m=+526.435772324" Nov 02 17:38:28.916549 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:28.916514 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-2899ab-59796778-drqbh"] Nov 02 17:38:28.920140 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:28.920125 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2899ab-59796778-drqbh" Nov 02 17:38:28.945792 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:28.945698 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-2899ab-serving-cert\"" Nov 02 17:38:28.949929 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:28.949901 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-2899ab-59796778-drqbh"] Nov 02 17:38:28.967491 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:28.967458 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f849213-a9f5-4236-ba98-a41e70d67dc5-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-2899ab-59796778-drqbh\" (UID: \"3f849213-a9f5-4236-ba98-a41e70d67dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2899ab-59796778-drqbh" Nov 02 17:38:28.967630 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:28.967505 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9tjm\" (UniqueName: \"kubernetes.io/projected/3f849213-a9f5-4236-ba98-a41e70d67dc5-kube-api-access-d9tjm\") pod \"maas-api-e2e-worker-w4-2899ab-59796778-drqbh\" (UID: \"3f849213-a9f5-4236-ba98-a41e70d67dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2899ab-59796778-drqbh" Nov 02 17:38:28.967774 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:28.967748 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f849213-a9f5-4236-ba98-a41e70d67dc5-maas-api-tls\") pod \"maas-api-e2e-worker-w4-2899ab-59796778-drqbh\" (UID: \"3f849213-a9f5-4236-ba98-a41e70d67dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2899ab-59796778-drqbh" Nov 02 17:38:28.967832 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:28.967817 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f849213-a9f5-4236-ba98-a41e70d67dc5-metrics-certs\") pod \"maas-api-e2e-worker-w4-2899ab-59796778-drqbh\" (UID: \"3f849213-a9f5-4236-ba98-a41e70d67dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2899ab-59796778-drqbh" Nov 02 17:38:29.068728 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:29.068692 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f849213-a9f5-4236-ba98-a41e70d67dc5-maas-api-tls\") pod \"maas-api-e2e-worker-w4-2899ab-59796778-drqbh\" (UID: \"3f849213-a9f5-4236-ba98-a41e70d67dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2899ab-59796778-drqbh" Nov 02 17:38:29.068902 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:29.068765 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f849213-a9f5-4236-ba98-a41e70d67dc5-metrics-certs\") pod \"maas-api-e2e-worker-w4-2899ab-59796778-drqbh\" (UID: \"3f849213-a9f5-4236-ba98-a41e70d67dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2899ab-59796778-drqbh" Nov 02 17:38:29.068902 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:29.068827 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f849213-a9f5-4236-ba98-a41e70d67dc5-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-2899ab-59796778-drqbh\" (UID: \"3f849213-a9f5-4236-ba98-a41e70d67dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2899ab-59796778-drqbh" Nov 02 17:38:29.068902 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:29.068852 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9tjm\" (UniqueName: \"kubernetes.io/projected/3f849213-a9f5-4236-ba98-a41e70d67dc5-kube-api-access-d9tjm\") pod \"maas-api-e2e-worker-w4-2899ab-59796778-drqbh\" (UID: \"3f849213-a9f5-4236-ba98-a41e70d67dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2899ab-59796778-drqbh" Nov 02 17:38:29.070021 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:29.069996 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f849213-a9f5-4236-ba98-a41e70d67dc5-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-2899ab-59796778-drqbh\" (UID: \"3f849213-a9f5-4236-ba98-a41e70d67dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2899ab-59796778-drqbh" Nov 02 17:38:29.071789 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:29.071762 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f849213-a9f5-4236-ba98-a41e70d67dc5-maas-api-tls\") pod \"maas-api-e2e-worker-w4-2899ab-59796778-drqbh\" (UID: \"3f849213-a9f5-4236-ba98-a41e70d67dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2899ab-59796778-drqbh" Nov 02 17:38:29.071972 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:29.071848 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f849213-a9f5-4236-ba98-a41e70d67dc5-metrics-certs\") pod \"maas-api-e2e-worker-w4-2899ab-59796778-drqbh\" (UID: \"3f849213-a9f5-4236-ba98-a41e70d67dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2899ab-59796778-drqbh" Nov 02 17:38:29.095648 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:29.095617 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9tjm\" (UniqueName: \"kubernetes.io/projected/3f849213-a9f5-4236-ba98-a41e70d67dc5-kube-api-access-d9tjm\") pod \"maas-api-e2e-worker-w4-2899ab-59796778-drqbh\" (UID: \"3f849213-a9f5-4236-ba98-a41e70d67dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2899ab-59796778-drqbh" Nov 02 17:38:29.237438 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:29.237358 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2899ab-59796778-drqbh" Nov 02 17:38:29.560175 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:38:29.560147 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f849213_a9f5_4236_ba98_a41e70d67dc5.slice/crio-99b917bdc4405764774567b2a03e216b02ca118e3cfbd715adda0f1dc5bddc31 WatchSource:0}: Error finding container 99b917bdc4405764774567b2a03e216b02ca118e3cfbd715adda0f1dc5bddc31: Status 404 returned error can't find the container with id 99b917bdc4405764774567b2a03e216b02ca118e3cfbd715adda0f1dc5bddc31 Nov 02 17:38:29.560517 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:29.560492 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-2899ab-59796778-drqbh"] Nov 02 17:38:29.698448 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:29.698411 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2899ab-59796778-drqbh" event={"ID":"3f849213-a9f5-4236-ba98-a41e70d67dc5","Type":"ContainerStarted","Data":"99b917bdc4405764774567b2a03e216b02ca118e3cfbd715adda0f1dc5bddc31"} Nov 02 17:38:30.678653 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:30.678620 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-120394-854c949567-5xfqn" Nov 02 17:38:30.703735 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:30.703703 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2899ab-59796778-drqbh" event={"ID":"3f849213-a9f5-4236-ba98-a41e70d67dc5","Type":"ContainerStarted","Data":"8bfb140074f3f645a0eb964263edf59bd150a3f532d6006c7ed43f1c15254cbb"} Nov 02 17:38:30.704103 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:30.703774 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2899ab-59796778-drqbh" Nov 02 17:38:30.821502 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:30.821444 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2899ab-59796778-drqbh" podStartSLOduration=2.513262202 podStartE2EDuration="2.821425077s" totalImagesPullingTime="308.162875ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:30.785286227 +0000 UTC m=+528.480641718" watchObservedRunningTime="2026-11-02 17:38:30.821425077 +0000 UTC m=+528.516780552" Nov 02 17:38:30.823825 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:30.823801 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb"] Nov 02 17:38:30.827513 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:30.827494 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb" Nov 02 17:38:30.848392 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:30.848362 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb"] Nov 02 17:38:30.848975 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:30.848948 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-5eedab-serving-cert\"" Nov 02 17:38:30.891129 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:30.891097 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5bf24bf-3188-425c-bced-4e77490d6377-maas-api-tls\") pod \"maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb\" (UID: \"c5bf24bf-3188-425c-bced-4e77490d6377\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb" Nov 02 17:38:30.891284 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:30.891155 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5bf24bf-3188-425c-bced-4e77490d6377-metrics-certs\") pod \"maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb\" (UID: \"c5bf24bf-3188-425c-bced-4e77490d6377\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb" Nov 02 17:38:30.891352 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:30.891300 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmhtt\" (UniqueName: \"kubernetes.io/projected/c5bf24bf-3188-425c-bced-4e77490d6377-kube-api-access-bmhtt\") pod \"maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb\" (UID: \"c5bf24bf-3188-425c-bced-4e77490d6377\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb" Nov 02 17:38:30.891446 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:30.891421 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5bf24bf-3188-425c-bced-4e77490d6377-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb\" (UID: \"c5bf24bf-3188-425c-bced-4e77490d6377\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb" Nov 02 17:38:30.993023 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:30.992923 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5bf24bf-3188-425c-bced-4e77490d6377-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb\" (UID: \"c5bf24bf-3188-425c-bced-4e77490d6377\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb" Nov 02 17:38:30.993187 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:30.993084 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5bf24bf-3188-425c-bced-4e77490d6377-maas-api-tls\") pod \"maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb\" (UID: \"c5bf24bf-3188-425c-bced-4e77490d6377\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb" Nov 02 17:38:30.993187 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:30.993138 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5bf24bf-3188-425c-bced-4e77490d6377-metrics-certs\") pod \"maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb\" (UID: \"c5bf24bf-3188-425c-bced-4e77490d6377\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb" Nov 02 17:38:30.993348 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:30.993209 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmhtt\" (UniqueName: \"kubernetes.io/projected/c5bf24bf-3188-425c-bced-4e77490d6377-kube-api-access-bmhtt\") pod \"maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb\" (UID: \"c5bf24bf-3188-425c-bced-4e77490d6377\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb" Nov 02 17:38:30.993830 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:30.993800 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5bf24bf-3188-425c-bced-4e77490d6377-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb\" (UID: \"c5bf24bf-3188-425c-bced-4e77490d6377\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb" Nov 02 17:38:30.996409 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:30.996382 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5bf24bf-3188-425c-bced-4e77490d6377-maas-api-tls\") pod \"maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb\" (UID: \"c5bf24bf-3188-425c-bced-4e77490d6377\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb" Nov 02 17:38:30.996840 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:30.996805 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5bf24bf-3188-425c-bced-4e77490d6377-metrics-certs\") pod \"maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb\" (UID: \"c5bf24bf-3188-425c-bced-4e77490d6377\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb" Nov 02 17:38:31.018394 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:31.018358 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmhtt\" (UniqueName: \"kubernetes.io/projected/c5bf24bf-3188-425c-bced-4e77490d6377-kube-api-access-bmhtt\") pod \"maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb\" (UID: \"c5bf24bf-3188-425c-bced-4e77490d6377\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb" Nov 02 17:38:31.143098 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:31.142912 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb" Nov 02 17:38:31.483190 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:38:31.483165 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5bf24bf_3188_425c_bced_4e77490d6377.slice/crio-f9b618f5202ec85f7b167761ac5655d4e8e1ae4a2d7f92e18c4c427d86128290 WatchSource:0}: Error finding container f9b618f5202ec85f7b167761ac5655d4e8e1ae4a2d7f92e18c4c427d86128290: Status 404 returned error can't find the container with id f9b618f5202ec85f7b167761ac5655d4e8e1ae4a2d7f92e18c4c427d86128290 Nov 02 17:38:31.484091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:31.484064 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb"] Nov 02 17:38:31.709800 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:31.709761 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb" event={"ID":"c5bf24bf-3188-425c-bced-4e77490d6377","Type":"ContainerStarted","Data":"f9b618f5202ec85f7b167761ac5655d4e8e1ae4a2d7f92e18c4c427d86128290"} Nov 02 17:38:32.716090 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:32.716046 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb" event={"ID":"c5bf24bf-3188-425c-bced-4e77490d6377","Type":"ContainerStarted","Data":"6e333e4b1477287274e52483305b3ca94097b6e76232be4049723c5ea6c421fc"} Nov 02 17:38:32.716547 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:32.716123 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb" Nov 02 17:38:32.771373 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:32.771327 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb" podStartSLOduration=2.391537589 podStartE2EDuration="2.771301339s" totalImagesPullingTime="379.76375ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:32.769217026 +0000 UTC m=+530.464572516" watchObservedRunningTime="2026-11-02 17:38:32.771301339 +0000 UTC m=+530.466656812" Nov 02 17:38:32.803087 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:32.803062 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:38:34.704647 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:34.704610 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6" Nov 02 17:38:36.715766 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:36.715735 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2899ab-59796778-drqbh" Nov 02 17:38:38.687723 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:38.687687 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4"] Nov 02 17:38:38.692681 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:38.692661 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4" Nov 02 17:38:38.729955 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:38.729926 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb" Nov 02 17:38:38.736727 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:38.736707 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-069fa5-serving-cert\"" Nov 02 17:38:38.738714 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:38.738696 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:38:38.784751 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:38.784718 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4"] Nov 02 17:38:38.806241 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:38.806199 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d53fefb9-19f7-4b4b-adec-75325b8ff619-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4\" (UID: \"d53fefb9-19f7-4b4b-adec-75325b8ff619\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4" Nov 02 17:38:38.806423 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:38.806333 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d53fefb9-19f7-4b4b-adec-75325b8ff619-metrics-certs\") pod \"maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4\" (UID: \"d53fefb9-19f7-4b4b-adec-75325b8ff619\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4" Nov 02 17:38:38.806423 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:38.806375 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d53fefb9-19f7-4b4b-adec-75325b8ff619-maas-api-tls\") pod \"maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4\" (UID: \"d53fefb9-19f7-4b4b-adec-75325b8ff619\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4" Nov 02 17:38:38.806423 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:38.806404 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5pdk\" (UniqueName: \"kubernetes.io/projected/d53fefb9-19f7-4b4b-adec-75325b8ff619-kube-api-access-f5pdk\") pod \"maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4\" (UID: \"d53fefb9-19f7-4b4b-adec-75325b8ff619\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4" Nov 02 17:38:38.907336 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:38.907289 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d53fefb9-19f7-4b4b-adec-75325b8ff619-metrics-certs\") pod \"maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4\" (UID: \"d53fefb9-19f7-4b4b-adec-75325b8ff619\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4" Nov 02 17:38:38.907523 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:38.907363 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d53fefb9-19f7-4b4b-adec-75325b8ff619-maas-api-tls\") pod \"maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4\" (UID: \"d53fefb9-19f7-4b4b-adec-75325b8ff619\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4" Nov 02 17:38:38.907523 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:38.907393 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5pdk\" (UniqueName: \"kubernetes.io/projected/d53fefb9-19f7-4b4b-adec-75325b8ff619-kube-api-access-f5pdk\") pod \"maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4\" (UID: \"d53fefb9-19f7-4b4b-adec-75325b8ff619\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4" Nov 02 17:38:38.907652 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:38.907616 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d53fefb9-19f7-4b4b-adec-75325b8ff619-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4\" (UID: \"d53fefb9-19f7-4b4b-adec-75325b8ff619\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4" Nov 02 17:38:38.908644 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:38.908615 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d53fefb9-19f7-4b4b-adec-75325b8ff619-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4\" (UID: \"d53fefb9-19f7-4b4b-adec-75325b8ff619\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4" Nov 02 17:38:38.911336 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:38.910643 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d53fefb9-19f7-4b4b-adec-75325b8ff619-metrics-certs\") pod \"maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4\" (UID: \"d53fefb9-19f7-4b4b-adec-75325b8ff619\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4" Nov 02 17:38:38.911336 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:38.910790 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d53fefb9-19f7-4b4b-adec-75325b8ff619-maas-api-tls\") pod \"maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4\" (UID: \"d53fefb9-19f7-4b4b-adec-75325b8ff619\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4" Nov 02 17:38:38.934929 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:38.934896 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5pdk\" (UniqueName: \"kubernetes.io/projected/d53fefb9-19f7-4b4b-adec-75325b8ff619-kube-api-access-f5pdk\") pod \"maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4\" (UID: \"d53fefb9-19f7-4b4b-adec-75325b8ff619\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4" Nov 02 17:38:39.007213 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:39.007130 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4" Nov 02 17:38:39.194511 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:38:39.194475 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:38:39.373896 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:39.373865 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4"] Nov 02 17:38:39.374133 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:38:39.374114 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd53fefb9_19f7_4b4b_adec_75325b8ff619.slice/crio-38f51a3faadf1b82c43f6d2c66f479bb68031bce60c3f3f9d154556d0adfdeae WatchSource:0}: Error finding container 38f51a3faadf1b82c43f6d2c66f479bb68031bce60c3f3f9d154556d0adfdeae: Status 404 returned error can't find the container with id 38f51a3faadf1b82c43f6d2c66f479bb68031bce60c3f3f9d154556d0adfdeae Nov 02 17:38:39.761326 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:39.756808 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4" event={"ID":"d53fefb9-19f7-4b4b-adec-75325b8ff619","Type":"ContainerStarted","Data":"38f51a3faadf1b82c43f6d2c66f479bb68031bce60c3f3f9d154556d0adfdeae"} Nov 02 17:38:40.789331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:40.785838 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4" event={"ID":"d53fefb9-19f7-4b4b-adec-75325b8ff619","Type":"ContainerStarted","Data":"094cf74c47a211c8eccf7301f8518af875010b7dc3ba3b08b2077e4ac2050133"} Nov 02 17:38:40.789331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:40.786819 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4" Nov 02 17:38:40.896331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:40.895700 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4" podStartSLOduration=2.537690634 podStartE2EDuration="2.895681241s" totalImagesPullingTime="357.990607ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:40.889669882 +0000 UTC m=+538.585025369" watchObservedRunningTime="2026-11-02 17:38:40.895681241 +0000 UTC m=+538.591036716" Nov 02 17:38:45.831260 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:45.831223 4471 generic.go:356] "Generic (PLEG): container finished" podID="006cd169-c635-4f97-a8d1-c21aa6c077c3" containerID="366bef4b47ff7c4e99b8ebe59103d9643b3e031789c9362d72ef563b40aff5c5" exitCode=0 Nov 02 17:38:45.831717 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:45.831293 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" event={"ID":"006cd169-c635-4f97-a8d1-c21aa6c077c3","Type":"ContainerDied","Data":"366bef4b47ff7c4e99b8ebe59103d9643b3e031789c9362d72ef563b40aff5c5"} Nov 02 17:38:45.831764 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:45.831740 4471 scope.go:122] "RemoveContainer" containerID="366bef4b47ff7c4e99b8ebe59103d9643b3e031789c9362d72ef563b40aff5c5" Nov 02 17:38:45.833428 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:45.833397 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:38:45.833559 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:45.833472 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:38:45.833559 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:45.833523 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:38:46.838839 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:46.838802 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" event={"ID":"006cd169-c635-4f97-a8d1-c21aa6c077c3","Type":"ContainerStarted","Data":"1f99e421a570c14824d51c908d12e950fa87c327183f1da649ed36b3c285c53d"} Nov 02 17:38:47.399502 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:47.399462 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:47.400915 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:47.400884 4471 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 17:38:47.400979 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:47.400953 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" podUID="006cd169-c635-4f97-a8d1-c21aa6c077c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 17:38:47.817498 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:47.817468 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4" Nov 02 17:38:48.399484 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:48.399446 4471 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 17:38:48.400021 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:48.399497 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" podUID="006cd169-c635-4f97-a8d1-c21aa6c077c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 17:38:48.633379 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:48.633331 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:49.295510 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:49.295477 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": read tcp 10.133.0.2:44476->10.133.0.40:15021: read: connection reset by peer" start-of-body= Nov 02 17:38:49.295678 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:49.295544 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" podUID="49caa894-b067-4d85-8141-741e72c8f041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": read tcp 10.133.0.2:44476->10.133.0.40:15021: read: connection reset by peer" Nov 02 17:38:49.343879 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:49.343834 4471 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": read tcp 10.133.0.2:47468->10.133.0.39:15021: read: connection reset by peer" start-of-body= Nov 02 17:38:49.344018 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:49.343915 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" podUID="14703ce4-5dc1-43d0-8368-c217f0bab9eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": read tcp 10.133.0.2:47468->10.133.0.39:15021: read: connection reset by peer" Nov 02 17:38:49.399029 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:49.398993 4471 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 17:38:49.399164 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:49.399051 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" podUID="006cd169-c635-4f97-a8d1-c21aa6c077c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 17:38:49.884063 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:49.881679 4471 generic.go:356] "Generic (PLEG): container finished" podID="49caa894-b067-4d85-8141-741e72c8f041" containerID="1d452628e1a1a514cdc498bf3e85994af2e768ec805be2df240146fedfe81cbe" exitCode=0 Nov 02 17:38:49.884063 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:49.881911 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" event={"ID":"49caa894-b067-4d85-8141-741e72c8f041","Type":"ContainerDied","Data":"1d452628e1a1a514cdc498bf3e85994af2e768ec805be2df240146fedfe81cbe"} Nov 02 17:38:49.884063 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:49.882573 4471 scope.go:122] "RemoveContainer" containerID="1d452628e1a1a514cdc498bf3e85994af2e768ec805be2df240146fedfe81cbe" Nov 02 17:38:49.894953 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:49.889568 4471 generic.go:356] "Generic (PLEG): container finished" podID="14703ce4-5dc1-43d0-8368-c217f0bab9eb" containerID="1972f28c65a6080b0770ac769426a03f8ef87c57fca1079e1a53948cfd1550de" exitCode=0 Nov 02 17:38:49.894953 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:49.889671 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" event={"ID":"14703ce4-5dc1-43d0-8368-c217f0bab9eb","Type":"ContainerDied","Data":"1972f28c65a6080b0770ac769426a03f8ef87c57fca1079e1a53948cfd1550de"} Nov 02 17:38:49.894953 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:49.890123 4471 scope.go:122] "RemoveContainer" containerID="1972f28c65a6080b0770ac769426a03f8ef87c57fca1079e1a53948cfd1550de" Nov 02 17:38:49.896265 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:49.896223 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:38:49.896545 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:49.896519 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:38:49.896694 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:49.896676 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:38:49.898321 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:49.898284 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:38:49.898420 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:49.898373 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:38:49.898490 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:49.898425 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:38:50.398835 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:50.398801 4471 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 17:38:50.399130 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:50.399097 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" podUID="006cd169-c635-4f97-a8d1-c21aa6c077c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 17:38:50.911336 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:50.909207 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" event={"ID":"49caa894-b067-4d85-8141-741e72c8f041","Type":"ContainerStarted","Data":"722539c79b3ec3867dbaaf17bb6298974482efc261c1181334e204b6499f5779"} Nov 02 17:38:50.911336 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:50.910225 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:50.927297 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:50.926161 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" event={"ID":"14703ce4-5dc1-43d0-8368-c217f0bab9eb","Type":"ContainerStarted","Data":"511355fc6cf5ddf746fdefe2a807368b2ab2c7cc5af36725e04d94cd9ca38119"} Nov 02 17:38:51.401329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:51.400711 4471 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 17:38:51.401329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:51.400777 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" podUID="006cd169-c635-4f97-a8d1-c21aa6c077c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 17:38:51.520331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:51.519765 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:38:51.709815 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:51.709726 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:51.710970 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:51.710881 4471 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 17:38:51.711190 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:51.711167 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" podUID="14703ce4-5dc1-43d0-8368-c217f0bab9eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 17:38:51.914783 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:51.914744 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 17:38:51.915412 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:51.915380 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" podUID="49caa894-b067-4d85-8141-741e72c8f041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 17:38:52.604328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:52.601671 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:52.619917 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:52.618616 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:38:52.713108 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:52.713077 4471 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 17:38:52.713435 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:52.713405 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" podUID="14703ce4-5dc1-43d0-8368-c217f0bab9eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 17:38:52.909744 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:52.909650 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 17:38:52.909744 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:52.909712 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" podUID="49caa894-b067-4d85-8141-741e72c8f041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 17:38:53.709727 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:53.708860 4471 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 17:38:53.709727 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:53.708945 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" podUID="14703ce4-5dc1-43d0-8368-c217f0bab9eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 17:38:53.909570 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:53.909525 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 17:38:53.909748 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:53.909594 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" podUID="49caa894-b067-4d85-8141-741e72c8f041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 17:38:54.708644 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:54.708594 4471 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 17:38:54.708813 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:54.708677 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" podUID="14703ce4-5dc1-43d0-8368-c217f0bab9eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 17:38:54.910353 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:54.909739 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 17:38:54.910353 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:54.909826 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" podUID="49caa894-b067-4d85-8141-741e72c8f041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 17:38:55.712334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:55.710924 4471 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 17:38:55.712334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:55.710996 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" podUID="14703ce4-5dc1-43d0-8368-c217f0bab9eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 17:38:55.910329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:55.909924 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 17:38:55.910329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:55.909990 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" podUID="49caa894-b067-4d85-8141-741e72c8f041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 17:38:56.716510 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:56.716479 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:56.716777 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:56.716752 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:56.717558 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:56.717532 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:38:56.909627 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:56.909595 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 17:38:56.909939 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:56.909897 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" podUID="49caa894-b067-4d85-8141-741e72c8f041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 17:38:57.912923 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:57.912889 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:57.913387 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:57.913186 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:38:57.913880 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:38:57.913862 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 17:39:05.761726 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:05.761694 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:39:09.182205 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:39:09.182168 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:39:13.555404 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:13.555362 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745"] Nov 02 17:39:13.625373 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:13.625338 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:39:13.629671 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:13.629644 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745"] Nov 02 17:39:13.654321 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:13.654282 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/520fe9ff-603f-4df8-8b71-06ccfdd1019e-tls-certs\") pod \"simulator-w2-f0a3b8-kserve-56df57c5cc-7m745\" (UID: \"520fe9ff-603f-4df8-8b71-06ccfdd1019e\") " pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:39:13.654438 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:13.654408 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/520fe9ff-603f-4df8-8b71-06ccfdd1019e-dshm\") pod \"simulator-w2-f0a3b8-kserve-56df57c5cc-7m745\" (UID: \"520fe9ff-603f-4df8-8b71-06ccfdd1019e\") " pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:39:13.654438 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:13.654434 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/520fe9ff-603f-4df8-8b71-06ccfdd1019e-model-cache\") pod \"simulator-w2-f0a3b8-kserve-56df57c5cc-7m745\" (UID: \"520fe9ff-603f-4df8-8b71-06ccfdd1019e\") " pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:39:13.654542 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:13.654523 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/520fe9ff-603f-4df8-8b71-06ccfdd1019e-tmp-dir\") pod \"simulator-w2-f0a3b8-kserve-56df57c5cc-7m745\" (UID: \"520fe9ff-603f-4df8-8b71-06ccfdd1019e\") " pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:39:13.654578 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:13.654551 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhk4z\" (UniqueName: \"kubernetes.io/projected/520fe9ff-603f-4df8-8b71-06ccfdd1019e-kube-api-access-xhk4z\") pod \"simulator-w2-f0a3b8-kserve-56df57c5cc-7m745\" (UID: \"520fe9ff-603f-4df8-8b71-06ccfdd1019e\") " pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:39:13.654678 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:13.654666 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/520fe9ff-603f-4df8-8b71-06ccfdd1019e-home\") pod \"simulator-w2-f0a3b8-kserve-56df57c5cc-7m745\" (UID: \"520fe9ff-603f-4df8-8b71-06ccfdd1019e\") " pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:39:13.656830 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:13.656785 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-f0a3b8\"/\"openshift-service-ca.crt\"" Nov 02 17:39:13.657178 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:13.657140 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-f0a3b8\"/\"kube-root-ca.crt\"" Nov 02 17:39:13.657337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:13.657320 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f0a3b8\"/\"default-dockercfg-4zkbn\"" Nov 02 17:39:13.657487 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:13.657472 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f0a3b8\"/\"simulator-w2-f0a3b8-kserve-self-signed-certs\"" Nov 02 17:39:13.756000 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:13.755967 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/520fe9ff-603f-4df8-8b71-06ccfdd1019e-dshm\") pod \"simulator-w2-f0a3b8-kserve-56df57c5cc-7m745\" (UID: \"520fe9ff-603f-4df8-8b71-06ccfdd1019e\") " pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:39:13.756000 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:13.756007 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/520fe9ff-603f-4df8-8b71-06ccfdd1019e-model-cache\") pod \"simulator-w2-f0a3b8-kserve-56df57c5cc-7m745\" (UID: \"520fe9ff-603f-4df8-8b71-06ccfdd1019e\") " pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:39:13.756269 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:13.756046 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/520fe9ff-603f-4df8-8b71-06ccfdd1019e-tmp-dir\") pod \"simulator-w2-f0a3b8-kserve-56df57c5cc-7m745\" (UID: \"520fe9ff-603f-4df8-8b71-06ccfdd1019e\") " pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:39:13.756269 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:13.756069 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhk4z\" (UniqueName: \"kubernetes.io/projected/520fe9ff-603f-4df8-8b71-06ccfdd1019e-kube-api-access-xhk4z\") pod \"simulator-w2-f0a3b8-kserve-56df57c5cc-7m745\" (UID: \"520fe9ff-603f-4df8-8b71-06ccfdd1019e\") " pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:39:13.756269 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:13.756173 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/520fe9ff-603f-4df8-8b71-06ccfdd1019e-home\") pod \"simulator-w2-f0a3b8-kserve-56df57c5cc-7m745\" (UID: \"520fe9ff-603f-4df8-8b71-06ccfdd1019e\") " pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:39:13.756269 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:13.756244 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/520fe9ff-603f-4df8-8b71-06ccfdd1019e-tls-certs\") pod \"simulator-w2-f0a3b8-kserve-56df57c5cc-7m745\" (UID: \"520fe9ff-603f-4df8-8b71-06ccfdd1019e\") " pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:39:13.756843 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:13.756812 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/520fe9ff-603f-4df8-8b71-06ccfdd1019e-tmp-dir\") pod \"simulator-w2-f0a3b8-kserve-56df57c5cc-7m745\" (UID: \"520fe9ff-603f-4df8-8b71-06ccfdd1019e\") " pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:39:13.756982 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:13.756824 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/520fe9ff-603f-4df8-8b71-06ccfdd1019e-home\") pod \"simulator-w2-f0a3b8-kserve-56df57c5cc-7m745\" (UID: \"520fe9ff-603f-4df8-8b71-06ccfdd1019e\") " pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:39:13.757440 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:13.757418 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/520fe9ff-603f-4df8-8b71-06ccfdd1019e-model-cache\") pod \"simulator-w2-f0a3b8-kserve-56df57c5cc-7m745\" (UID: \"520fe9ff-603f-4df8-8b71-06ccfdd1019e\") " pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:39:13.759392 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:13.759355 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/520fe9ff-603f-4df8-8b71-06ccfdd1019e-tls-certs\") pod \"simulator-w2-f0a3b8-kserve-56df57c5cc-7m745\" (UID: \"520fe9ff-603f-4df8-8b71-06ccfdd1019e\") " pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:39:13.762717 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:13.762697 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/520fe9ff-603f-4df8-8b71-06ccfdd1019e-dshm\") pod \"simulator-w2-f0a3b8-kserve-56df57c5cc-7m745\" (UID: \"520fe9ff-603f-4df8-8b71-06ccfdd1019e\") " pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:39:13.780234 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:13.780205 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhk4z\" (UniqueName: \"kubernetes.io/projected/520fe9ff-603f-4df8-8b71-06ccfdd1019e-kube-api-access-xhk4z\") pod \"simulator-w2-f0a3b8-kserve-56df57c5cc-7m745\" (UID: \"520fe9ff-603f-4df8-8b71-06ccfdd1019e\") " pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:39:13.940980 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:13.940896 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:39:14.239284 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:14.239244 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745"] Nov 02 17:39:14.239449 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:39:14.239370 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod520fe9ff_603f_4df8_8b71_06ccfdd1019e.slice/crio-76b4422da004c196cf173b84b9e3727964fb76f436906eb8fae4d34abbddeece WatchSource:0}: Error finding container 76b4422da004c196cf173b84b9e3727964fb76f436906eb8fae4d34abbddeece: Status 404 returned error can't find the container with id 76b4422da004c196cf173b84b9e3727964fb76f436906eb8fae4d34abbddeece Nov 02 17:39:15.076723 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:15.076685 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" event={"ID":"520fe9ff-603f-4df8-8b71-06ccfdd1019e","Type":"ContainerStarted","Data":"b8c969d3404243c9d12216bd17d9562b58249fcd9522d82e7a11ed8a2fdcf7d4"} Nov 02 17:39:15.077088 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:15.076737 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" event={"ID":"520fe9ff-603f-4df8-8b71-06ccfdd1019e","Type":"ContainerStarted","Data":"76b4422da004c196cf173b84b9e3727964fb76f436906eb8fae4d34abbddeece"} Nov 02 17:39:15.119269 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:15.119221 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podStartSLOduration=2.119206312 podStartE2EDuration="2.119206312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:15.11868118 +0000 UTC m=+572.814036672" watchObservedRunningTime="2026-11-02 17:39:15.119206312 +0000 UTC m=+572.814561789" Nov 02 17:39:15.941674 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:15.941630 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:39:20.095501 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:20.095464 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:39:23.941865 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:23.941821 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:39:23.954979 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:23.954952 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:39:24.131656 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:24.131625 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:39:26.167142 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:26.167055 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2"] Nov 02 17:39:26.170471 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:26.170452 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:39:26.189427 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:26.189399 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-7df526\"/\"kube-root-ca.crt\"" Nov 02 17:39:26.202252 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:26.202222 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7df526\"/\"premium-w6-7df526-kserve-self-signed-certs\"" Nov 02 17:39:26.202252 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:26.202244 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-7df526\"/\"openshift-service-ca.crt\"" Nov 02 17:39:26.202496 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:26.202467 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7df526\"/\"default-dockercfg-gs46w\"" Nov 02 17:39:26.204700 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:26.204678 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2"] Nov 02 17:39:26.290680 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:26.290643 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/537a8e6f-da32-4e42-a7a1-bd7c061eabea-tls-certs\") pod \"premium-w6-7df526-kserve-57bd79fd6b-6vnm2\" (UID: \"537a8e6f-da32-4e42-a7a1-bd7c061eabea\") " pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:39:26.290886 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:26.290697 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/537a8e6f-da32-4e42-a7a1-bd7c061eabea-tmp-dir\") pod \"premium-w6-7df526-kserve-57bd79fd6b-6vnm2\" (UID: \"537a8e6f-da32-4e42-a7a1-bd7c061eabea\") " pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:39:26.290886 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:26.290787 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/537a8e6f-da32-4e42-a7a1-bd7c061eabea-dshm\") pod \"premium-w6-7df526-kserve-57bd79fd6b-6vnm2\" (UID: \"537a8e6f-da32-4e42-a7a1-bd7c061eabea\") " pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:39:26.291009 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:26.290890 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvwft\" (UniqueName: \"kubernetes.io/projected/537a8e6f-da32-4e42-a7a1-bd7c061eabea-kube-api-access-nvwft\") pod \"premium-w6-7df526-kserve-57bd79fd6b-6vnm2\" (UID: \"537a8e6f-da32-4e42-a7a1-bd7c061eabea\") " pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:39:26.291009 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:26.290966 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/537a8e6f-da32-4e42-a7a1-bd7c061eabea-home\") pod \"premium-w6-7df526-kserve-57bd79fd6b-6vnm2\" (UID: \"537a8e6f-da32-4e42-a7a1-bd7c061eabea\") " pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:39:26.291009 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:26.290992 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/537a8e6f-da32-4e42-a7a1-bd7c061eabea-model-cache\") pod \"premium-w6-7df526-kserve-57bd79fd6b-6vnm2\" (UID: \"537a8e6f-da32-4e42-a7a1-bd7c061eabea\") " pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:39:26.391885 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:26.391847 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/537a8e6f-da32-4e42-a7a1-bd7c061eabea-tls-certs\") pod \"premium-w6-7df526-kserve-57bd79fd6b-6vnm2\" (UID: \"537a8e6f-da32-4e42-a7a1-bd7c061eabea\") " pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:39:26.392091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:26.391909 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/537a8e6f-da32-4e42-a7a1-bd7c061eabea-tmp-dir\") pod \"premium-w6-7df526-kserve-57bd79fd6b-6vnm2\" (UID: \"537a8e6f-da32-4e42-a7a1-bd7c061eabea\") " pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:39:26.392091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:26.391952 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/537a8e6f-da32-4e42-a7a1-bd7c061eabea-dshm\") pod \"premium-w6-7df526-kserve-57bd79fd6b-6vnm2\" (UID: \"537a8e6f-da32-4e42-a7a1-bd7c061eabea\") " pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:39:26.392091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:26.392001 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvwft\" (UniqueName: \"kubernetes.io/projected/537a8e6f-da32-4e42-a7a1-bd7c061eabea-kube-api-access-nvwft\") pod \"premium-w6-7df526-kserve-57bd79fd6b-6vnm2\" (UID: \"537a8e6f-da32-4e42-a7a1-bd7c061eabea\") " pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:39:26.392091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:26.392044 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/537a8e6f-da32-4e42-a7a1-bd7c061eabea-home\") pod \"premium-w6-7df526-kserve-57bd79fd6b-6vnm2\" (UID: \"537a8e6f-da32-4e42-a7a1-bd7c061eabea\") " pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:39:26.392091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:26.392058 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/537a8e6f-da32-4e42-a7a1-bd7c061eabea-model-cache\") pod \"premium-w6-7df526-kserve-57bd79fd6b-6vnm2\" (UID: \"537a8e6f-da32-4e42-a7a1-bd7c061eabea\") " pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:39:26.392398 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:26.392372 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/537a8e6f-da32-4e42-a7a1-bd7c061eabea-tmp-dir\") pod \"premium-w6-7df526-kserve-57bd79fd6b-6vnm2\" (UID: \"537a8e6f-da32-4e42-a7a1-bd7c061eabea\") " pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:39:26.392537 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:26.392513 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/537a8e6f-da32-4e42-a7a1-bd7c061eabea-home\") pod \"premium-w6-7df526-kserve-57bd79fd6b-6vnm2\" (UID: \"537a8e6f-da32-4e42-a7a1-bd7c061eabea\") " pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:39:26.392591 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:26.392532 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/537a8e6f-da32-4e42-a7a1-bd7c061eabea-model-cache\") pod \"premium-w6-7df526-kserve-57bd79fd6b-6vnm2\" (UID: \"537a8e6f-da32-4e42-a7a1-bd7c061eabea\") " pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:39:26.394293 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:26.394269 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/537a8e6f-da32-4e42-a7a1-bd7c061eabea-dshm\") pod \"premium-w6-7df526-kserve-57bd79fd6b-6vnm2\" (UID: \"537a8e6f-da32-4e42-a7a1-bd7c061eabea\") " pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:39:26.394691 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:26.394670 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/537a8e6f-da32-4e42-a7a1-bd7c061eabea-tls-certs\") pod \"premium-w6-7df526-kserve-57bd79fd6b-6vnm2\" (UID: \"537a8e6f-da32-4e42-a7a1-bd7c061eabea\") " pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:39:26.416626 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:26.416599 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvwft\" (UniqueName: \"kubernetes.io/projected/537a8e6f-da32-4e42-a7a1-bd7c061eabea-kube-api-access-nvwft\") pod \"premium-w6-7df526-kserve-57bd79fd6b-6vnm2\" (UID: \"537a8e6f-da32-4e42-a7a1-bd7c061eabea\") " pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:39:26.482630 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:26.482556 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:39:26.798394 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:26.798355 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2"] Nov 02 17:39:26.798609 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:39:26.798574 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod537a8e6f_da32_4e42_a7a1_bd7c061eabea.slice/crio-42e8332285812657ae7deb0d48bbf3dc4ba63f6d9f06d315356b7b6fb7a8ef5a WatchSource:0}: Error finding container 42e8332285812657ae7deb0d48bbf3dc4ba63f6d9f06d315356b7b6fb7a8ef5a: Status 404 returned error can't find the container with id 42e8332285812657ae7deb0d48bbf3dc4ba63f6d9f06d315356b7b6fb7a8ef5a Nov 02 17:39:27.136243 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:27.136158 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" event={"ID":"537a8e6f-da32-4e42-a7a1-bd7c061eabea","Type":"ContainerStarted","Data":"40366d0d43189fab94fb9070497b79fc4a1bceaf15279f58aaaff3220b485f13"} Nov 02 17:39:27.136243 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:27.136199 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" event={"ID":"537a8e6f-da32-4e42-a7a1-bd7c061eabea","Type":"ContainerStarted","Data":"42e8332285812657ae7deb0d48bbf3dc4ba63f6d9f06d315356b7b6fb7a8ef5a"} Nov 02 17:39:27.182455 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:27.182406 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podStartSLOduration=1.182393242 podStartE2EDuration="1.182393242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:39:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:27.18107615 +0000 UTC m=+584.876431636" watchObservedRunningTime="2026-11-02 17:39:27.182393242 +0000 UTC m=+584.877748715" Nov 02 17:39:27.483297 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:27.483211 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:39:36.482918 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:36.482882 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:39:36.495445 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:36.495418 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:39:37.214358 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:37.211328 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:39:39.182403 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:39:39.182369 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:39:40.971171 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:40.971134 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:39:47.099079 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:47.099043 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d"] Nov 02 17:39:47.102971 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:47.102947 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:39:47.134199 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:47.134164 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-863b87\"/\"kube-root-ca.crt\"" Nov 02 17:39:47.134199 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:47.134165 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-863b87\"/\"openshift-service-ca.crt\"" Nov 02 17:39:47.134199 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:47.134191 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-863b87\"/\"default-dockercfg-h47km\"" Nov 02 17:39:47.134461 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:47.134266 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-863b87\"/\"premium-w5-863b87-kserve-self-signed-certs\"" Nov 02 17:39:47.137928 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:47.137904 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d"] Nov 02 17:39:47.229102 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:47.229071 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e729ed8-8407-477c-90a7-c899203f2452-dshm\") pod \"premium-w5-863b87-kserve-cfb8948c5-t2x8d\" (UID: \"9e729ed8-8407-477c-90a7-c899203f2452\") " pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:39:47.229263 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:47.229115 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e729ed8-8407-477c-90a7-c899203f2452-model-cache\") pod \"premium-w5-863b87-kserve-cfb8948c5-t2x8d\" (UID: \"9e729ed8-8407-477c-90a7-c899203f2452\") " pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:39:47.229263 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:47.229213 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e729ed8-8407-477c-90a7-c899203f2452-home\") pod \"premium-w5-863b87-kserve-cfb8948c5-t2x8d\" (UID: \"9e729ed8-8407-477c-90a7-c899203f2452\") " pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:39:47.229263 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:47.229253 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svtt8\" (UniqueName: \"kubernetes.io/projected/9e729ed8-8407-477c-90a7-c899203f2452-kube-api-access-svtt8\") pod \"premium-w5-863b87-kserve-cfb8948c5-t2x8d\" (UID: \"9e729ed8-8407-477c-90a7-c899203f2452\") " pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:39:47.229414 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:47.229378 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e729ed8-8407-477c-90a7-c899203f2452-tls-certs\") pod \"premium-w5-863b87-kserve-cfb8948c5-t2x8d\" (UID: \"9e729ed8-8407-477c-90a7-c899203f2452\") " pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:39:47.229457 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:47.229420 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e729ed8-8407-477c-90a7-c899203f2452-tmp-dir\") pod \"premium-w5-863b87-kserve-cfb8948c5-t2x8d\" (UID: \"9e729ed8-8407-477c-90a7-c899203f2452\") " pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:39:47.330555 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:47.330524 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e729ed8-8407-477c-90a7-c899203f2452-dshm\") pod \"premium-w5-863b87-kserve-cfb8948c5-t2x8d\" (UID: \"9e729ed8-8407-477c-90a7-c899203f2452\") " pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:39:47.330736 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:47.330574 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e729ed8-8407-477c-90a7-c899203f2452-model-cache\") pod \"premium-w5-863b87-kserve-cfb8948c5-t2x8d\" (UID: \"9e729ed8-8407-477c-90a7-c899203f2452\") " pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:39:47.330736 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:47.330613 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e729ed8-8407-477c-90a7-c899203f2452-home\") pod \"premium-w5-863b87-kserve-cfb8948c5-t2x8d\" (UID: \"9e729ed8-8407-477c-90a7-c899203f2452\") " pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:39:47.330736 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:47.330643 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svtt8\" (UniqueName: \"kubernetes.io/projected/9e729ed8-8407-477c-90a7-c899203f2452-kube-api-access-svtt8\") pod \"premium-w5-863b87-kserve-cfb8948c5-t2x8d\" (UID: \"9e729ed8-8407-477c-90a7-c899203f2452\") " pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:39:47.330736 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:47.330698 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e729ed8-8407-477c-90a7-c899203f2452-tls-certs\") pod \"premium-w5-863b87-kserve-cfb8948c5-t2x8d\" (UID: \"9e729ed8-8407-477c-90a7-c899203f2452\") " pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:39:47.330736 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:47.330735 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e729ed8-8407-477c-90a7-c899203f2452-tmp-dir\") pod \"premium-w5-863b87-kserve-cfb8948c5-t2x8d\" (UID: \"9e729ed8-8407-477c-90a7-c899203f2452\") " pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:39:47.331076 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:47.331039 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e729ed8-8407-477c-90a7-c899203f2452-home\") pod \"premium-w5-863b87-kserve-cfb8948c5-t2x8d\" (UID: \"9e729ed8-8407-477c-90a7-c899203f2452\") " pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:39:47.331076 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:47.331053 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e729ed8-8407-477c-90a7-c899203f2452-model-cache\") pod \"premium-w5-863b87-kserve-cfb8948c5-t2x8d\" (UID: \"9e729ed8-8407-477c-90a7-c899203f2452\") " pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:39:47.331179 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:47.331104 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e729ed8-8407-477c-90a7-c899203f2452-tmp-dir\") pod \"premium-w5-863b87-kserve-cfb8948c5-t2x8d\" (UID: \"9e729ed8-8407-477c-90a7-c899203f2452\") " pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:39:47.332948 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:47.332927 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e729ed8-8407-477c-90a7-c899203f2452-dshm\") pod \"premium-w5-863b87-kserve-cfb8948c5-t2x8d\" (UID: \"9e729ed8-8407-477c-90a7-c899203f2452\") " pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:39:47.333167 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:47.333151 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e729ed8-8407-477c-90a7-c899203f2452-tls-certs\") pod \"premium-w5-863b87-kserve-cfb8948c5-t2x8d\" (UID: \"9e729ed8-8407-477c-90a7-c899203f2452\") " pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:39:47.355028 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:47.354962 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svtt8\" (UniqueName: \"kubernetes.io/projected/9e729ed8-8407-477c-90a7-c899203f2452-kube-api-access-svtt8\") pod \"premium-w5-863b87-kserve-cfb8948c5-t2x8d\" (UID: \"9e729ed8-8407-477c-90a7-c899203f2452\") " pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:39:47.414547 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:47.414523 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:39:47.711749 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:47.711721 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d"] Nov 02 17:39:47.712088 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:39:47.712058 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e729ed8_8407_477c_90a7_c899203f2452.slice/crio-19940231b65540a954ceb42998284bbf43949a37eee22860434fe737cee80857 WatchSource:0}: Error finding container 19940231b65540a954ceb42998284bbf43949a37eee22860434fe737cee80857: Status 404 returned error can't find the container with id 19940231b65540a954ceb42998284bbf43949a37eee22860434fe737cee80857 Nov 02 17:39:48.254067 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:48.254024 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" event={"ID":"9e729ed8-8407-477c-90a7-c899203f2452","Type":"ContainerStarted","Data":"56ae181916890cc38e68e75e804f52118d0ca1f6441c80a08fff91b0552615ed"} Nov 02 17:39:48.254476 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:48.254077 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" event={"ID":"9e729ed8-8407-477c-90a7-c899203f2452","Type":"ContainerStarted","Data":"19940231b65540a954ceb42998284bbf43949a37eee22860434fe737cee80857"} Nov 02 17:39:48.297682 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:48.297633 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podStartSLOduration=1.297619302 podStartE2EDuration="1.297619302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:48.297134992 +0000 UTC m=+605.992490465" watchObservedRunningTime="2026-11-02 17:39:48.297619302 +0000 UTC m=+605.992974776" Nov 02 17:39:48.415044 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:48.415011 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:39:57.415739 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:57.415700 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:39:57.428488 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:57.428464 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:39:58.316812 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:39:58.316776 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:40:00.400201 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.400165 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg"] Nov 02 17:40:00.403263 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.403247 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:40:00.431429 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.431398 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-7df526-openshift-default-dockercfg-cjrwk\"" Nov 02 17:40:00.444740 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.444714 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg"] Nov 02 17:40:00.478113 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.478078 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85637eda-7caa-4bc0-b697-1acf7f1da917-istio-podinfo\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:40:00.478282 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.478140 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-istio-data\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:40:00.478282 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.478258 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-workload-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:40:00.478405 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.478299 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-credential-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:40:00.478405 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.478378 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-workload-certs\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:40:00.478494 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.478419 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85637eda-7caa-4bc0-b697-1acf7f1da917-istio-token\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:40:00.478494 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.478454 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-istio-envoy\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:40:00.478494 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.478470 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85637eda-7caa-4bc0-b697-1acf7f1da917-istiod-ca-cert\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:40:00.478640 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.478617 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z6pt\" (UniqueName: \"kubernetes.io/projected/85637eda-7caa-4bc0-b697-1acf7f1da917-kube-api-access-6z6pt\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:40:00.560069 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.560036 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll"] Nov 02 17:40:00.562713 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.562696 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 17:40:00.579833 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.579807 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-workload-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:40:00.579960 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.579855 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-credential-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:40:00.580009 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.579987 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-workload-certs\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:40:00.580103 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.580078 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85637eda-7caa-4bc0-b697-1acf7f1da917-istio-token\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:40:00.580255 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.580231 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-istio-envoy\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:40:00.580356 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.580274 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85637eda-7caa-4bc0-b697-1acf7f1da917-istiod-ca-cert\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:40:00.580356 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.580232 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-workload-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:40:00.580356 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.580330 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z6pt\" (UniqueName: \"kubernetes.io/projected/85637eda-7caa-4bc0-b697-1acf7f1da917-kube-api-access-6z6pt\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:40:00.580528 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.580403 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85637eda-7caa-4bc0-b697-1acf7f1da917-istio-podinfo\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:40:00.580528 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.580466 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-workload-certs\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:40:00.580528 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.580487 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-istio-data\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:40:00.580695 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.580668 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-credential-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:40:00.580873 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.580850 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-istio-data\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:40:00.580941 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.580929 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85637eda-7caa-4bc0-b697-1acf7f1da917-istiod-ca-cert\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:40:00.582445 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.582424 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-istio-envoy\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:40:00.582762 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.582745 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85637eda-7caa-4bc0-b697-1acf7f1da917-istio-podinfo\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:40:00.584468 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.584446 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-120394-openshift-default-dockercfg-qq496\"" Nov 02 17:40:00.590210 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.590185 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll"] Nov 02 17:40:00.607940 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.607906 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85637eda-7caa-4bc0-b697-1acf7f1da917-istio-token\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:40:00.608137 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.608118 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z6pt\" (UniqueName: \"kubernetes.io/projected/85637eda-7caa-4bc0-b697-1acf7f1da917-kube-api-access-6z6pt\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:40:00.681583 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.681503 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istio-data\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 17:40:00.681583 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.681541 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istiod-ca-cert\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 17:40:00.681583 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.681565 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istio-token\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 17:40:00.681817 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.681719 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istio-envoy\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 17:40:00.681817 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.681759 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-credential-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 17:40:00.681909 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.681839 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmnll\" (UniqueName: \"kubernetes.io/projected/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-kube-api-access-bmnll\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 17:40:00.681950 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.681915 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-workload-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 17:40:00.681950 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.681936 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istio-podinfo\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 17:40:00.682033 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.681971 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-workload-certs\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 17:40:00.716560 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.716529 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:40:00.783218 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.783177 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-workload-certs\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 17:40:00.783383 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.783261 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istio-data\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 17:40:00.783383 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.783291 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istiod-ca-cert\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 17:40:00.783383 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.783358 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istio-token\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 17:40:00.783626 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.783437 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istio-envoy\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 17:40:00.783626 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.783491 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-credential-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 17:40:00.783626 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.783558 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmnll\" (UniqueName: \"kubernetes.io/projected/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-kube-api-access-bmnll\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 17:40:00.783786 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.783649 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-workload-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 17:40:00.783786 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.783672 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istio-podinfo\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 17:40:00.785070 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.785041 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istio-data\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 17:40:00.785403 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.785382 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-credential-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 17:40:00.786025 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.786000 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-workload-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 17:40:00.786812 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.786786 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-workload-certs\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 17:40:00.787151 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.787123 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istiod-ca-cert\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 17:40:00.787711 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.787692 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istio-envoy\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 17:40:00.791370 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.791350 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istio-podinfo\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 17:40:00.808153 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.808119 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istio-token\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 17:40:00.808691 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.808672 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmnll\" (UniqueName: \"kubernetes.io/projected/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-kube-api-access-bmnll\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 17:40:00.875135 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:00.875108 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 17:40:01.030339 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:40:01.030255 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85637eda_7caa_4bc0_b697_1acf7f1da917.slice/crio-47d77fac58e0ffc25db7b7a1f37107ddf77ff76a88e46c4fb39489a1880afd0a WatchSource:0}: Error finding container 47d77fac58e0ffc25db7b7a1f37107ddf77ff76a88e46c4fb39489a1880afd0a: Status 404 returned error can't find the container with id 47d77fac58e0ffc25db7b7a1f37107ddf77ff76a88e46c4fb39489a1880afd0a Nov 02 17:40:01.031761 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:01.031737 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg"] Nov 02 17:40:01.033258 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:01.033216 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:40:01.033394 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:01.033300 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:40:01.033394 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:01.033359 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:40:01.184965 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:01.184938 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll"] Nov 02 17:40:01.185906 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:40:01.185868 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636cf7f7_ba1a_4f54_9db0_6d8ad2909424.slice/crio-da81a82f5e2f15d2aae61fd8db3b18131a83e5c35a49fbe60ef6d5f6585cd921 WatchSource:0}: Error finding container da81a82f5e2f15d2aae61fd8db3b18131a83e5c35a49fbe60ef6d5f6585cd921: Status 404 returned error can't find the container with id da81a82f5e2f15d2aae61fd8db3b18131a83e5c35a49fbe60ef6d5f6585cd921 Nov 02 17:40:01.188581 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:01.188528 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:40:01.188674 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:01.188641 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:40:01.188711 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:01.188694 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:40:01.323706 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:01.323605 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" event={"ID":"85637eda-7caa-4bc0-b697-1acf7f1da917","Type":"ContainerStarted","Data":"e0df35339cd046c1b1729013cf72507ab8d8f5c36842e269aba20629304b77b8"} Nov 02 17:40:01.323706 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:01.323668 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" event={"ID":"85637eda-7caa-4bc0-b697-1acf7f1da917","Type":"ContainerStarted","Data":"47d77fac58e0ffc25db7b7a1f37107ddf77ff76a88e46c4fb39489a1880afd0a"} Nov 02 17:40:01.325642 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:01.325612 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" event={"ID":"636cf7f7-ba1a-4f54-9db0-6d8ad2909424","Type":"ContainerStarted","Data":"08ff90ee8d0ceace53f4f33083f4104ad89d3c9d5e2d23fec1ec3401760de26d"} Nov 02 17:40:01.325873 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:01.325663 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" event={"ID":"636cf7f7-ba1a-4f54-9db0-6d8ad2909424","Type":"ContainerStarted","Data":"da81a82f5e2f15d2aae61fd8db3b18131a83e5c35a49fbe60ef6d5f6585cd921"} Nov 02 17:40:01.377464 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:01.377403 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" podStartSLOduration=1.377383211 podStartE2EDuration="1.377383211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:01.375616658 +0000 UTC m=+619.070972143" watchObservedRunningTime="2026-11-02 17:40:01.377383211 +0000 UTC m=+619.072738685" Nov 02 17:40:01.427202 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:01.427150 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" podStartSLOduration=1.42713529 podStartE2EDuration="1.42713529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:01.424950506 +0000 UTC m=+619.120306000" watchObservedRunningTime="2026-11-02 17:40:01.42713529 +0000 UTC m=+619.122490762" Nov 02 17:40:01.717711 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:01.717631 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:40:01.879442 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:01.879396 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 17:40:02.718746 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:02.718710 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 02 17:40:02.719128 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:02.718764 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" podUID="85637eda-7caa-4bc0-b697-1acf7f1da917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 02 17:40:02.878329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:02.877440 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 17:40:02.878329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:02.877504 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" podUID="636cf7f7-ba1a-4f54-9db0-6d8ad2909424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 17:40:03.717827 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:03.717789 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 02 17:40:03.718005 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:03.717846 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" podUID="85637eda-7caa-4bc0-b697-1acf7f1da917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 02 17:40:03.883329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:03.877544 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 17:40:03.883329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:03.877609 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" podUID="636cf7f7-ba1a-4f54-9db0-6d8ad2909424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 17:40:04.717483 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:04.717450 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 02 17:40:04.717652 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:04.717503 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" podUID="85637eda-7caa-4bc0-b697-1acf7f1da917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 02 17:40:04.879330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:04.879230 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 17:40:04.879330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:04.879322 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" podUID="636cf7f7-ba1a-4f54-9db0-6d8ad2909424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 17:40:05.717332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:05.717272 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 02 17:40:05.717789 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:05.717762 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" podUID="85637eda-7caa-4bc0-b697-1acf7f1da917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 02 17:40:05.883329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:05.879789 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 17:40:05.883329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:05.879855 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" podUID="636cf7f7-ba1a-4f54-9db0-6d8ad2909424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 17:40:06.717834 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:06.717795 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 02 17:40:06.718411 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:06.717868 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" podUID="85637eda-7caa-4bc0-b697-1acf7f1da917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 02 17:40:06.879339 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:06.878664 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 17:40:06.879339 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:06.878732 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" podUID="636cf7f7-ba1a-4f54-9db0-6d8ad2909424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 17:40:07.823853 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:07.823821 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:40:07.824774 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:07.824749 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:40:07.826819 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:07.825994 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:40:08.004631 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:08.004602 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 17:40:08.004976 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:08.004950 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 17:40:08.009895 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:08.009872 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 17:40:09.182414 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:40:09.182376 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:40:10.481779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:10.481742 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:40:31.223761 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.223723 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv"] Nov 02 17:40:31.228316 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.228275 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 17:40:31.255680 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.255651 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-069fa5-openshift-default-dockercfg-ckxtj\"" Nov 02 17:40:31.279564 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.279527 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv"] Nov 02 17:40:31.374688 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.374639 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istio-data\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 17:40:31.374985 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.374958 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istio-token\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 17:40:31.377288 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.375110 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istio-envoy\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 17:40:31.377597 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.377561 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-credential-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 17:40:31.377836 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.377813 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-workload-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 17:40:31.377973 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.377958 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlqvs\" (UniqueName: \"kubernetes.io/projected/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-kube-api-access-rlqvs\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 17:40:31.378216 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.378115 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-workload-certs\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 17:40:31.378412 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.378393 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istio-podinfo\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 17:40:31.378553 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.378536 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istiod-ca-cert\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 17:40:31.479361 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.479251 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-workload-certs\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 17:40:31.479361 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.479322 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istio-podinfo\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 17:40:31.479361 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.479356 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istiod-ca-cert\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 17:40:31.479649 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.479430 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istio-data\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 17:40:31.479649 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.479529 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istio-token\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 17:40:31.479649 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.479582 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istio-envoy\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 17:40:31.479810 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.479773 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-credential-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 17:40:31.480073 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.480044 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-workload-certs\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 17:40:31.480154 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.480103 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istiod-ca-cert\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 17:40:31.480214 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.480159 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istio-data\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 17:40:31.480214 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.480104 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-credential-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 17:40:31.480214 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.480201 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-workload-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 17:40:31.480401 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.480246 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlqvs\" (UniqueName: \"kubernetes.io/projected/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-kube-api-access-rlqvs\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 17:40:31.480573 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.480550 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-workload-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 17:40:31.482069 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.482039 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istio-envoy\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 17:40:31.483531 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.483507 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istio-podinfo\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 17:40:31.503707 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.503681 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istio-token\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 17:40:31.561780 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.560069 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlqvs\" (UniqueName: \"kubernetes.io/projected/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-kube-api-access-rlqvs\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 17:40:31.561780 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.561554 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 17:40:31.947936 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:40:31.947899 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c572702_1be9_42ec_a4b7_243fe8bfd9c8.slice/crio-1d5acf3eba4dd03b9b7c7f9781dfe656bae31b3e8f1694e52946ce6b84e0c6e0 WatchSource:0}: Error finding container 1d5acf3eba4dd03b9b7c7f9781dfe656bae31b3e8f1694e52946ce6b84e0c6e0: Status 404 returned error can't find the container with id 1d5acf3eba4dd03b9b7c7f9781dfe656bae31b3e8f1694e52946ce6b84e0c6e0 Nov 02 17:40:31.950059 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.950034 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv"] Nov 02 17:40:31.950517 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.950485 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:40:31.950594 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.950554 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:40:31.950645 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:31.950604 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:40:32.575630 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:32.575588 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" event={"ID":"9c572702-1be9-42ec-a4b7-243fe8bfd9c8","Type":"ContainerStarted","Data":"d0a03b0dd74fe6c8c960e59fa859d7379f67979ef8a98fa11419476cfa87702d"} Nov 02 17:40:32.576248 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:32.576212 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" event={"ID":"9c572702-1be9-42ec-a4b7-243fe8bfd9c8","Type":"ContainerStarted","Data":"1d5acf3eba4dd03b9b7c7f9781dfe656bae31b3e8f1694e52946ce6b84e0c6e0"} Nov 02 17:40:32.682330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:32.673954 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" podStartSLOduration=1.673936375 podStartE2EDuration="1.673936375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:32.6498465 +0000 UTC m=+650.345202029" watchObservedRunningTime="2026-11-02 17:40:32.673936375 +0000 UTC m=+650.369291848" Nov 02 17:40:33.584332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:33.575340 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 17:40:33.591532 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:33.591110 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 02 17:40:33.591532 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:33.591185 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" podUID="9c572702-1be9-42ec-a4b7-243fe8bfd9c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 02 17:40:34.565865 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:34.565827 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 02 17:40:34.566155 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:34.566125 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" podUID="9c572702-1be9-42ec-a4b7-243fe8bfd9c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 02 17:40:35.574383 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:35.573664 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 02 17:40:35.574383 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:35.573742 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" podUID="9c572702-1be9-42ec-a4b7-243fe8bfd9c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 02 17:40:36.567331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:36.564416 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 02 17:40:36.567331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:36.564481 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" podUID="9c572702-1be9-42ec-a4b7-243fe8bfd9c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 02 17:40:37.572718 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:37.572677 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 02 17:40:37.573336 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:37.573284 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" podUID="9c572702-1be9-42ec-a4b7-243fe8bfd9c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 02 17:40:38.562796 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:38.562756 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 02 17:40:38.563079 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:38.562837 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" podUID="9c572702-1be9-42ec-a4b7-243fe8bfd9c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 02 17:40:39.182793 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:40:39.182755 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:40:39.567512 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:39.567484 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 17:40:39.567761 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:39.567739 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 17:40:39.568384 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:39.568365 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 17:40:47.260487 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:40:47.260452 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:41:00.615921 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:00.615830 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z"] Nov 02 17:41:00.619932 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:00.619910 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 17:41:00.702973 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:00.702892 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z"] Nov 02 17:41:00.794757 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:00.794703 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-workload-certs\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 17:41:00.794951 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:00.794824 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db524df0-282c-43eb-936b-53ed7d9388cf-istio-podinfo\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 17:41:00.794951 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:00.794922 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zlvt\" (UniqueName: \"kubernetes.io/projected/db524df0-282c-43eb-936b-53ed7d9388cf-kube-api-access-2zlvt\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 17:41:00.795097 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:00.795079 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-credential-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 17:41:00.795157 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:00.795119 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-istio-envoy\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 17:41:00.795157 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:00.795148 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-istio-data\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 17:41:00.795333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:00.795274 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db524df0-282c-43eb-936b-53ed7d9388cf-istiod-ca-cert\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 17:41:00.795400 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:00.795361 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-workload-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 17:41:00.795453 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:00.795404 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db524df0-282c-43eb-936b-53ed7d9388cf-istio-token\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 17:41:00.896207 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:00.896122 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-credential-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 17:41:00.896207 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:00.896182 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-istio-envoy\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 17:41:00.896496 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:00.896324 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-istio-data\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 17:41:00.896496 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:00.896397 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db524df0-282c-43eb-936b-53ed7d9388cf-istiod-ca-cert\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 17:41:00.896496 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:00.896425 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-workload-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 17:41:00.896496 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:00.896450 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db524df0-282c-43eb-936b-53ed7d9388cf-istio-token\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 17:41:00.896709 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:00.896548 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-workload-certs\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 17:41:00.896709 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:00.896586 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db524df0-282c-43eb-936b-53ed7d9388cf-istio-podinfo\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 17:41:00.896709 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:00.896612 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-credential-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 17:41:00.896709 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:00.896641 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zlvt\" (UniqueName: \"kubernetes.io/projected/db524df0-282c-43eb-936b-53ed7d9388cf-kube-api-access-2zlvt\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 17:41:00.896709 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:00.896690 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-istio-data\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 17:41:00.896963 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:00.896801 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-workload-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 17:41:00.896963 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:00.896860 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-workload-certs\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 17:41:00.897219 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:00.897196 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db524df0-282c-43eb-936b-53ed7d9388cf-istiod-ca-cert\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 17:41:00.899014 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:00.898982 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-istio-envoy\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 17:41:00.899115 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:00.899011 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db524df0-282c-43eb-936b-53ed7d9388cf-istio-podinfo\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 17:41:00.921134 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:00.921104 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db524df0-282c-43eb-936b-53ed7d9388cf-istio-token\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 17:41:00.921270 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:00.921251 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zlvt\" (UniqueName: \"kubernetes.io/projected/db524df0-282c-43eb-936b-53ed7d9388cf-kube-api-access-2zlvt\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 17:41:01.001123 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.001092 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 17:41:01.323782 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:41:01.323752 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb524df0_282c_43eb_936b_53ed7d9388cf.slice/crio-97d50e56be368bfa139ffc947eb40e86229fb8f4d2f5191af670f8d52cc09775 WatchSource:0}: Error finding container 97d50e56be368bfa139ffc947eb40e86229fb8f4d2f5191af670f8d52cc09775: Status 404 returned error can't find the container with id 97d50e56be368bfa139ffc947eb40e86229fb8f4d2f5191af670f8d52cc09775 Nov 02 17:41:01.324249 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.324224 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z"] Nov 02 17:41:01.326905 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.326867 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:41:01.327003 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.326984 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:41:01.327057 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.327033 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:41:01.345558 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.345533 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q"] Nov 02 17:41:01.350896 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.350879 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 17:41:01.367559 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.367540 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-f0a3b8-openshift-default-dockercfg-nch4k\"" Nov 02 17:41:01.375085 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.375065 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q"] Nov 02 17:41:01.503051 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.503008 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-istio-data\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 17:41:01.503231 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.503068 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-credential-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 17:41:01.503231 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.503216 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fae7257d-c6ad-4e86-8133-aacf8110adb5-istiod-ca-cert\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 17:41:01.503364 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.503342 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fae7257d-c6ad-4e86-8133-aacf8110adb5-istio-token\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 17:41:01.503422 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.503402 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fae7257d-c6ad-4e86-8133-aacf8110adb5-istio-podinfo\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 17:41:01.503517 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.503497 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-workload-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 17:41:01.503581 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.503564 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-istio-envoy\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 17:41:01.503670 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.503655 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcxnf\" (UniqueName: \"kubernetes.io/projected/fae7257d-c6ad-4e86-8133-aacf8110adb5-kube-api-access-zcxnf\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 17:41:01.503733 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.503687 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-workload-certs\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 17:41:01.605054 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.604963 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcxnf\" (UniqueName: \"kubernetes.io/projected/fae7257d-c6ad-4e86-8133-aacf8110adb5-kube-api-access-zcxnf\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 17:41:01.605054 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.605013 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-workload-certs\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 17:41:01.605294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.605062 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-istio-data\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 17:41:01.605294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.605109 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-credential-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 17:41:01.605435 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.605291 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fae7257d-c6ad-4e86-8133-aacf8110adb5-istiod-ca-cert\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 17:41:01.605435 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.605425 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fae7257d-c6ad-4e86-8133-aacf8110adb5-istio-token\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 17:41:01.605578 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.605555 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-istio-data\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 17:41:01.605646 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.605587 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-workload-certs\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 17:41:01.605646 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.605629 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fae7257d-c6ad-4e86-8133-aacf8110adb5-istio-podinfo\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 17:41:01.605752 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.605651 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-credential-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 17:41:01.605993 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.605971 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-workload-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 17:41:01.606128 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.606093 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-istio-envoy\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 17:41:01.606198 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.606158 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fae7257d-c6ad-4e86-8133-aacf8110adb5-istiod-ca-cert\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 17:41:01.606475 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.606443 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-workload-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 17:41:01.608292 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.608260 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fae7257d-c6ad-4e86-8133-aacf8110adb5-istio-podinfo\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 17:41:01.608422 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.608295 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-istio-envoy\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 17:41:01.632297 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.632267 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcxnf\" (UniqueName: \"kubernetes.io/projected/fae7257d-c6ad-4e86-8133-aacf8110adb5-kube-api-access-zcxnf\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 17:41:01.632681 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.632435 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fae7257d-c6ad-4e86-8133-aacf8110adb5-istio-token\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 17:41:01.669150 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.669113 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 17:41:01.780677 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.780645 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" event={"ID":"db524df0-282c-43eb-936b-53ed7d9388cf","Type":"ContainerStarted","Data":"233bdc170834e90343633263b943122af0ad834932a7aa1adccf6a682acc8239"} Nov 02 17:41:01.780836 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.780696 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" event={"ID":"db524df0-282c-43eb-936b-53ed7d9388cf","Type":"ContainerStarted","Data":"97d50e56be368bfa139ffc947eb40e86229fb8f4d2f5191af670f8d52cc09775"} Nov 02 17:41:01.832839 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.832788 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" podStartSLOduration=1.8327727280000001 podStartE2EDuration="1.832772728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:01.83159941 +0000 UTC m=+679.526954907" watchObservedRunningTime="2026-11-02 17:41:01.832772728 +0000 UTC m=+679.528128200" Nov 02 17:41:01.985252 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.984813 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q"] Nov 02 17:41:01.988091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.986358 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:41:01.988091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.986500 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:41:01.988091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:01.986588 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:41:02.001578 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:02.001549 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 17:41:02.003437 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:02.003412 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 02 17:41:02.003637 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:02.003610 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" podUID="db524df0-282c-43eb-936b-53ed7d9388cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 02 17:41:02.865335 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:02.855549 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" event={"ID":"fae7257d-c6ad-4e86-8133-aacf8110adb5","Type":"ContainerStarted","Data":"cae7f6af307bb2b5fad1524414c9317dafb1ac517d675cf661261dffb26772f2"} Nov 02 17:41:02.908540 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:02.908461 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" event={"ID":"fae7257d-c6ad-4e86-8133-aacf8110adb5","Type":"ContainerStarted","Data":"8192ecc877d1c61c98a83af1e31c123b0aba9847ef9e8f3a9d579c22ed8e23f1"} Nov 02 17:41:02.941033 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:02.940978 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" podStartSLOduration=1.9409637229999999 podStartE2EDuration="1.940963723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:02.940092831 +0000 UTC m=+680.635448304" watchObservedRunningTime="2026-11-02 17:41:02.940963723 +0000 UTC m=+680.636319268" Nov 02 17:41:03.002837 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:03.002792 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 02 17:41:03.003140 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:03.003107 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" podUID="db524df0-282c-43eb-936b-53ed7d9388cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 02 17:41:03.676421 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:03.676378 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 17:41:03.678465 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:03.678438 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 02 17:41:03.678668 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:03.678639 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" podUID="fae7257d-c6ad-4e86-8133-aacf8110adb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 02 17:41:04.004818 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:04.004732 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 02 17:41:04.005772 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:04.005362 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" podUID="db524df0-282c-43eb-936b-53ed7d9388cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 02 17:41:04.675329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:04.669659 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 02 17:41:04.675329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:04.669719 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" podUID="fae7257d-c6ad-4e86-8133-aacf8110adb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 02 17:41:05.005876 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:05.005777 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 02 17:41:05.005876 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:05.005844 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" podUID="db524df0-282c-43eb-936b-53ed7d9388cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 02 17:41:05.670123 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:05.669674 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 02 17:41:05.670123 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:05.669736 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" podUID="fae7257d-c6ad-4e86-8133-aacf8110adb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 02 17:41:06.004393 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:06.003679 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 02 17:41:06.004393 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:06.003744 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" podUID="db524df0-282c-43eb-936b-53ed7d9388cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 02 17:41:06.671724 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:06.671681 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 02 17:41:06.672361 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:06.672293 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" podUID="fae7257d-c6ad-4e86-8133-aacf8110adb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 02 17:41:07.004811 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:07.004704 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 02 17:41:07.004811 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:07.004772 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" podUID="db524df0-282c-43eb-936b-53ed7d9388cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 02 17:41:07.672327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:07.670671 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 02 17:41:07.672327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:07.670733 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" podUID="fae7257d-c6ad-4e86-8133-aacf8110adb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 02 17:41:08.001767 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:08.001680 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 02 17:41:08.001767 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:08.001756 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" podUID="db524df0-282c-43eb-936b-53ed7d9388cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 02 17:41:08.670185 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:08.670142 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 02 17:41:08.670429 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:08.670215 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" podUID="fae7257d-c6ad-4e86-8133-aacf8110adb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 02 17:41:09.006696 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:09.006609 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 17:41:09.007122 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:09.007047 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 17:41:09.007940 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:09.007916 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 17:41:09.188766 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:41:09.188722 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:41:09.675591 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:09.675560 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 17:41:09.675887 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:09.675860 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 17:41:09.676818 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:09.676799 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 17:41:14.025120 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:14.025086 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:41:39.182568 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:41:39.182526 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:41:44.284934 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:44.284900 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:41:56.711656 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:56.711624 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf"] Nov 02 17:41:56.729426 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:56.729397 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:41:56.734719 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:56.734692 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf"] Nov 02 17:41:56.757319 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:56.757280 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f0a3b8\"/\"distinct-2-w2-f0a3b8-kserve-self-signed-certs\"" Nov 02 17:41:56.846590 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:56.846550 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-home\") pod \"distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf\" (UID: \"abaa1ee3-4cc9-47d4-9c44-0ce826b1e496\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:41:56.846764 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:56.846664 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-tmp-dir\") pod \"distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf\" (UID: \"abaa1ee3-4cc9-47d4-9c44-0ce826b1e496\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:41:56.846764 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:56.846724 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-model-cache\") pod \"distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf\" (UID: \"abaa1ee3-4cc9-47d4-9c44-0ce826b1e496\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:41:56.846764 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:56.846758 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-dshm\") pod \"distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf\" (UID: \"abaa1ee3-4cc9-47d4-9c44-0ce826b1e496\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:41:56.846881 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:56.846859 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-tls-certs\") pod \"distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf\" (UID: \"abaa1ee3-4cc9-47d4-9c44-0ce826b1e496\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:41:56.846931 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:56.846917 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwnbz\" (UniqueName: \"kubernetes.io/projected/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-kube-api-access-zwnbz\") pod \"distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf\" (UID: \"abaa1ee3-4cc9-47d4-9c44-0ce826b1e496\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:41:56.950557 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:56.950025 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-tls-certs\") pod \"distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf\" (UID: \"abaa1ee3-4cc9-47d4-9c44-0ce826b1e496\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:41:56.950557 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:56.950130 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwnbz\" (UniqueName: \"kubernetes.io/projected/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-kube-api-access-zwnbz\") pod \"distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf\" (UID: \"abaa1ee3-4cc9-47d4-9c44-0ce826b1e496\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:41:56.950557 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:56.950209 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-home\") pod \"distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf\" (UID: \"abaa1ee3-4cc9-47d4-9c44-0ce826b1e496\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:41:56.950557 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:56.950299 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-tmp-dir\") pod \"distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf\" (UID: \"abaa1ee3-4cc9-47d4-9c44-0ce826b1e496\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:41:56.950557 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:56.950411 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-model-cache\") pod \"distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf\" (UID: \"abaa1ee3-4cc9-47d4-9c44-0ce826b1e496\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:41:56.950557 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:56.950461 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-dshm\") pod \"distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf\" (UID: \"abaa1ee3-4cc9-47d4-9c44-0ce826b1e496\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:41:56.951402 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:56.951371 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-tmp-dir\") pod \"distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf\" (UID: \"abaa1ee3-4cc9-47d4-9c44-0ce826b1e496\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:41:56.951632 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:56.951603 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-home\") pod \"distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf\" (UID: \"abaa1ee3-4cc9-47d4-9c44-0ce826b1e496\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:41:56.951906 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:56.951883 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-model-cache\") pod \"distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf\" (UID: \"abaa1ee3-4cc9-47d4-9c44-0ce826b1e496\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:41:56.970777 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:56.970699 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-dshm\") pod \"distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf\" (UID: \"abaa1ee3-4cc9-47d4-9c44-0ce826b1e496\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:41:56.981241 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:56.979759 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-tls-certs\") pod \"distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf\" (UID: \"abaa1ee3-4cc9-47d4-9c44-0ce826b1e496\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:41:56.983332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:56.983275 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwnbz\" (UniqueName: \"kubernetes.io/projected/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-kube-api-access-zwnbz\") pod \"distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf\" (UID: \"abaa1ee3-4cc9-47d4-9c44-0ce826b1e496\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:41:57.043038 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:57.043008 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:41:57.348751 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:57.348720 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf"] Nov 02 17:41:57.348864 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:41:57.348763 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabaa1ee3_4cc9_47d4_9c44_0ce826b1e496.slice/crio-e3cd802b95c42af44ec62411e293ac47e84204f1662317821140c4a8c851fba0 WatchSource:0}: Error finding container e3cd802b95c42af44ec62411e293ac47e84204f1662317821140c4a8c851fba0: Status 404 returned error can't find the container with id e3cd802b95c42af44ec62411e293ac47e84204f1662317821140c4a8c851fba0 Nov 02 17:41:58.350993 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:58.350956 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" event={"ID":"abaa1ee3-4cc9-47d4-9c44-0ce826b1e496","Type":"ContainerStarted","Data":"10f6d0d2a3460de32ba19c0328b3e18836c632df28c44794e9d80b16b8800151"} Nov 02 17:41:58.351559 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:58.351536 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" event={"ID":"abaa1ee3-4cc9-47d4-9c44-0ce826b1e496","Type":"ContainerStarted","Data":"e3cd802b95c42af44ec62411e293ac47e84204f1662317821140c4a8c851fba0"} Nov 02 17:41:58.393807 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:58.393755 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podStartSLOduration=2.393739412 podStartE2EDuration="2.393739412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:58.39329186 +0000 UTC m=+736.088647333" watchObservedRunningTime="2026-11-02 17:41:58.393739412 +0000 UTC m=+736.089094886" Nov 02 17:41:59.043859 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:41:59.043825 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:42:01.295887 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.295849 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd"] Nov 02 17:42:01.299508 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.299485 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:42:01.325341 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.325272 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd"] Nov 02 17:42:01.432192 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.432161 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-istio-envoy\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:42:01.432388 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.432198 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-credential-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:42:01.432388 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.432281 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec23f89c-765f-4c30-a3e5-761ca79f0288-istio-podinfo\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:42:01.432388 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.432329 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-workload-certs\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:42:01.432388 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.432374 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m95w2\" (UniqueName: \"kubernetes.io/projected/ec23f89c-765f-4c30-a3e5-761ca79f0288-kube-api-access-m95w2\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:42:01.432543 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.432491 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec23f89c-765f-4c30-a3e5-761ca79f0288-istio-token\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:42:01.432578 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.432549 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-workload-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:42:01.432608 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.432590 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-istio-data\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:42:01.432655 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.432642 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec23f89c-765f-4c30-a3e5-761ca79f0288-istiod-ca-cert\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:42:01.535170 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.534138 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-workload-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:42:01.535170 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.534188 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-istio-data\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:42:01.535170 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.534229 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec23f89c-765f-4c30-a3e5-761ca79f0288-istiod-ca-cert\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:42:01.535170 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.534449 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-istio-envoy\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:42:01.535170 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.534491 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-credential-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:42:01.535170 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.534546 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec23f89c-765f-4c30-a3e5-761ca79f0288-istio-podinfo\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:42:01.535170 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.534584 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-workload-certs\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:42:01.535170 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.534655 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m95w2\" (UniqueName: \"kubernetes.io/projected/ec23f89c-765f-4c30-a3e5-761ca79f0288-kube-api-access-m95w2\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:42:01.535170 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.534749 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec23f89c-765f-4c30-a3e5-761ca79f0288-istio-token\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:42:01.538063 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.536279 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-workload-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:42:01.538063 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.536566 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-istio-data\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:42:01.538063 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.537142 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec23f89c-765f-4c30-a3e5-761ca79f0288-istiod-ca-cert\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:42:01.541271 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.540652 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-istio-envoy\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:42:01.541271 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.540908 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-credential-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:42:01.542109 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.542084 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-workload-certs\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:42:01.543521 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.543496 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec23f89c-765f-4c30-a3e5-761ca79f0288-istio-podinfo\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:42:01.565762 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.565702 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec23f89c-765f-4c30-a3e5-761ca79f0288-istio-token\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:42:01.565872 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.565853 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m95w2\" (UniqueName: \"kubernetes.io/projected/ec23f89c-765f-4c30-a3e5-761ca79f0288-kube-api-access-m95w2\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:42:01.613932 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.613897 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:42:01.927947 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:42:01.927914 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec23f89c_765f_4c30_a3e5_761ca79f0288.slice/crio-6bdc92b4504236612ff16a808f9d4f0388f4dcbea09ba2443b37e7666abfc677 WatchSource:0}: Error finding container 6bdc92b4504236612ff16a808f9d4f0388f4dcbea09ba2443b37e7666abfc677: Status 404 returned error can't find the container with id 6bdc92b4504236612ff16a808f9d4f0388f4dcbea09ba2443b37e7666abfc677 Nov 02 17:42:01.928707 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.928683 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd"] Nov 02 17:42:01.930462 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.930427 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:42:01.930569 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.930491 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:42:01.930569 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:01.930519 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:42:02.380391 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:02.380360 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" event={"ID":"ec23f89c-765f-4c30-a3e5-761ca79f0288","Type":"ContainerStarted","Data":"bc1f92a2701a88d7785bca48ccbed6f62a183cda50a47468e285924dc4e2a0ae"} Nov 02 17:42:02.380786 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:02.380419 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" event={"ID":"ec23f89c-765f-4c30-a3e5-761ca79f0288","Type":"ContainerStarted","Data":"6bdc92b4504236612ff16a808f9d4f0388f4dcbea09ba2443b37e7666abfc677"} Nov 02 17:42:02.429907 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:02.429855 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" podStartSLOduration=1.429841208 podStartE2EDuration="1.429841208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:02.427917004 +0000 UTC m=+740.123272478" watchObservedRunningTime="2026-11-02 17:42:02.429841208 +0000 UTC m=+740.125196681" Nov 02 17:42:02.614795 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:02.614758 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:42:03.615971 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:03.615939 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 17:42:03.616379 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:03.615989 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" podUID="ec23f89c-765f-4c30-a3e5-761ca79f0288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 17:42:04.614863 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:04.614834 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 17:42:04.615017 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:04.614883 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" podUID="ec23f89c-765f-4c30-a3e5-761ca79f0288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 17:42:05.614442 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:05.614406 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 17:42:05.614851 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:05.614463 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" podUID="ec23f89c-765f-4c30-a3e5-761ca79f0288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 17:42:06.614968 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:06.614934 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 17:42:06.615546 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:06.614989 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" podUID="ec23f89c-765f-4c30-a3e5-761ca79f0288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 17:42:07.043978 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:07.043947 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:42:07.072680 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:07.072647 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:42:07.418989 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:07.418897 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:42:07.623328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:07.620827 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 17:42:07.623328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:07.620891 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" podUID="ec23f89c-765f-4c30-a3e5-761ca79f0288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 17:42:08.614873 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:08.614834 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 17:42:08.615068 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:08.614893 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" podUID="ec23f89c-765f-4c30-a3e5-761ca79f0288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 17:42:09.183140 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:42:09.183102 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:42:09.618680 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:09.618650 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:42:09.619115 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:09.619094 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:42:09.619754 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:09.619734 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:42:15.208668 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:15.208633 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:42:15.931453 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:15.931402 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx"] Nov 02 17:42:16.013325 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.010743 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx"] Nov 02 17:42:16.013325 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.010854 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:42:16.055631 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.055559 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-120394\"/\"openshift-service-ca.crt\"" Nov 02 17:42:16.055631 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.055559 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-120394\"/\"kube-root-ca.crt\"" Nov 02 17:42:16.055948 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.055688 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-120394\"/\"distinct-w1-120394-kserve-self-signed-certs\"" Nov 02 17:42:16.055948 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.055732 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-120394\"/\"default-dockercfg-8x5zb\"" Nov 02 17:42:16.058391 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.058366 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286"] Nov 02 17:42:16.062072 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.062052 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:42:16.071623 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.071600 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-home\") pod \"distinct-w1-120394-kserve-65cf4d8b55-sxlkx\" (UID: \"6ce077ac-aaca-4afc-8090-2fb935b4a3e9\") " pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:42:16.071727 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.071713 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-model-cache\") pod \"distinct-w1-120394-kserve-65cf4d8b55-sxlkx\" (UID: \"6ce077ac-aaca-4afc-8090-2fb935b4a3e9\") " pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:42:16.071782 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.071734 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-tmp-dir\") pod \"distinct-w1-120394-kserve-65cf4d8b55-sxlkx\" (UID: \"6ce077ac-aaca-4afc-8090-2fb935b4a3e9\") " pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:42:16.071782 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.071762 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-tls-certs\") pod \"distinct-w1-120394-kserve-65cf4d8b55-sxlkx\" (UID: \"6ce077ac-aaca-4afc-8090-2fb935b4a3e9\") " pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:42:16.071979 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.071954 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58p85\" (UniqueName: \"kubernetes.io/projected/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-kube-api-access-58p85\") pod \"distinct-w1-120394-kserve-65cf4d8b55-sxlkx\" (UID: \"6ce077ac-aaca-4afc-8090-2fb935b4a3e9\") " pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:42:16.072079 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.072049 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-dshm\") pod \"distinct-w1-120394-kserve-65cf4d8b55-sxlkx\" (UID: \"6ce077ac-aaca-4afc-8090-2fb935b4a3e9\") " pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:42:16.080393 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.080363 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286"] Nov 02 17:42:16.173865 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.173822 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chgjx\" (UniqueName: \"kubernetes.io/projected/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-kube-api-access-chgjx\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9m286\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:42:16.174079 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.174017 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-home\") pod \"distinct-w1-120394-kserve-65cf4d8b55-sxlkx\" (UID: \"6ce077ac-aaca-4afc-8090-2fb935b4a3e9\") " pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:42:16.174211 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.174154 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-workload-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9m286\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:42:16.174278 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.174235 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istio-podinfo\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9m286\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:42:16.174434 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.174406 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-model-cache\") pod \"distinct-w1-120394-kserve-65cf4d8b55-sxlkx\" (UID: \"6ce077ac-aaca-4afc-8090-2fb935b4a3e9\") " pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:42:16.174505 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.174449 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-tmp-dir\") pod \"distinct-w1-120394-kserve-65cf4d8b55-sxlkx\" (UID: \"6ce077ac-aaca-4afc-8090-2fb935b4a3e9\") " pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:42:16.174505 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.174451 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-home\") pod \"distinct-w1-120394-kserve-65cf4d8b55-sxlkx\" (UID: \"6ce077ac-aaca-4afc-8090-2fb935b4a3e9\") " pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:42:16.174505 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.174478 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-credential-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9m286\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:42:16.174676 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.174511 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-tls-certs\") pod \"distinct-w1-120394-kserve-65cf4d8b55-sxlkx\" (UID: \"6ce077ac-aaca-4afc-8090-2fb935b4a3e9\") " pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:42:16.174676 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.174599 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-workload-certs\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9m286\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:42:16.174676 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.174624 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istio-token\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9m286\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:42:16.174823 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.174676 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58p85\" (UniqueName: \"kubernetes.io/projected/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-kube-api-access-58p85\") pod \"distinct-w1-120394-kserve-65cf4d8b55-sxlkx\" (UID: \"6ce077ac-aaca-4afc-8090-2fb935b4a3e9\") " pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:42:16.174823 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.174711 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-model-cache\") pod \"distinct-w1-120394-kserve-65cf4d8b55-sxlkx\" (UID: \"6ce077ac-aaca-4afc-8090-2fb935b4a3e9\") " pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:42:16.174823 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.174727 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-dshm\") pod \"distinct-w1-120394-kserve-65cf4d8b55-sxlkx\" (UID: \"6ce077ac-aaca-4afc-8090-2fb935b4a3e9\") " pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:42:16.174823 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.174757 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istio-data\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9m286\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:42:16.175021 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.174823 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istiod-ca-cert\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9m286\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:42:16.175021 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.174853 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istio-envoy\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9m286\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:42:16.175385 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.175355 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-tmp-dir\") pod \"distinct-w1-120394-kserve-65cf4d8b55-sxlkx\" (UID: \"6ce077ac-aaca-4afc-8090-2fb935b4a3e9\") " pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:42:16.177496 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.177473 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-dshm\") pod \"distinct-w1-120394-kserve-65cf4d8b55-sxlkx\" (UID: \"6ce077ac-aaca-4afc-8090-2fb935b4a3e9\") " pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:42:16.177768 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.177747 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-tls-certs\") pod \"distinct-w1-120394-kserve-65cf4d8b55-sxlkx\" (UID: \"6ce077ac-aaca-4afc-8090-2fb935b4a3e9\") " pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:42:16.200883 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.200797 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58p85\" (UniqueName: \"kubernetes.io/projected/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-kube-api-access-58p85\") pod \"distinct-w1-120394-kserve-65cf4d8b55-sxlkx\" (UID: \"6ce077ac-aaca-4afc-8090-2fb935b4a3e9\") " pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:42:16.279976 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.279938 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-workload-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9m286\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:42:16.280957 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.280934 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istio-podinfo\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9m286\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:42:16.281144 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.281128 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-credential-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9m286\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:42:16.281338 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.281299 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-workload-certs\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9m286\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:42:16.281456 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.281441 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istio-token\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9m286\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:42:16.281596 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.281584 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istio-data\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9m286\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:42:16.281729 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.281714 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istiod-ca-cert\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9m286\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:42:16.281822 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.281810 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istio-envoy\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9m286\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:42:16.281934 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.281920 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chgjx\" (UniqueName: \"kubernetes.io/projected/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-kube-api-access-chgjx\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9m286\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:42:16.282951 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.282925 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-workload-certs\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9m286\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:42:16.283102 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.280868 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-workload-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9m286\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:42:16.287127 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.285681 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-credential-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9m286\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:42:16.287127 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.285926 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istio-data\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9m286\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:42:16.287127 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.286741 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istiod-ca-cert\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9m286\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:42:16.301744 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.301677 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istio-envoy\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9m286\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:42:16.348981 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.348263 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:42:16.419321 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.419266 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istio-podinfo\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9m286\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:42:16.419979 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.419864 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chgjx\" (UniqueName: \"kubernetes.io/projected/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-kube-api-access-chgjx\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9m286\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:42:16.435651 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.435513 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istio-token\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-9m286\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:42:16.684257 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:16.679468 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:42:17.021364 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:17.021298 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx"] Nov 02 17:42:17.620391 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:17.604845 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" event={"ID":"6ce077ac-aaca-4afc-8090-2fb935b4a3e9","Type":"ContainerStarted","Data":"3f54cc0face5f04e55d57a5b630e58aa8e2f1511cb86631180ec0e50a4912619"} Nov 02 17:42:17.691461 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:17.691371 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286"] Nov 02 17:42:17.721138 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:17.721100 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:42:17.721398 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:17.721360 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:42:17.721950 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:17.721566 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:42:18.613522 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:18.613475 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" event={"ID":"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c","Type":"ContainerStarted","Data":"680fb021801780d34675eb6036f760230f7b7e1fe854273780e7d450df04379e"} Nov 02 17:42:18.613747 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:18.613542 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" event={"ID":"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c","Type":"ContainerStarted","Data":"9b2f213589a32f5db86d710e4739cdf6789cda71b7797b6f72dc3633f88a1c57"} Nov 02 17:42:18.615838 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:18.615806 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" event={"ID":"6ce077ac-aaca-4afc-8090-2fb935b4a3e9","Type":"ContainerStarted","Data":"7c3a2c8754db1a5d87cd2e50329239a0a6d6c02106e58a0a6f23d52e6c160f9b"} Nov 02 17:42:18.684327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:18.680257 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:42:18.684327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:18.683107 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-9m286 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 17:42:18.684327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:18.683256 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" podUID="65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 17:42:18.687510 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:18.687453 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" podStartSLOduration=2.687432918 podStartE2EDuration="2.687432918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:18.669818114 +0000 UTC m=+756.365173586" watchObservedRunningTime="2026-11-02 17:42:18.687432918 +0000 UTC m=+756.382788394" Nov 02 17:42:18.713988 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:18.713930 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podStartSLOduration=3.713915729 podStartE2EDuration="3.713915729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:18.712240859 +0000 UTC m=+756.407596333" watchObservedRunningTime="2026-11-02 17:42:18.713915729 +0000 UTC m=+756.409271201" Nov 02 17:42:19.349422 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:19.349387 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:42:19.680863 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:19.680769 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-9m286 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 17:42:19.680863 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:19.680841 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" podUID="65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 17:42:20.681407 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:20.681370 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-9m286 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 17:42:20.681839 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:20.681427 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" podUID="65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 17:42:21.680710 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:21.680678 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-9m286 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 17:42:21.680885 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:21.680731 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" podUID="65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 17:42:22.680279 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:22.680243 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-9m286 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 17:42:22.680911 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:22.680867 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" podUID="65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 17:42:23.680665 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:23.680624 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-9m286 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 17:42:23.681123 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:23.680687 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" podUID="65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 17:42:24.859170 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:24.859079 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:42:24.860622 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:24.860596 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:42:24.861767 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:24.861749 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:42:26.349745 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:26.349711 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:42:26.362700 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:26.362671 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:42:26.680342 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:26.680233 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:42:37.068296 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:37.068255 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j"] Nov 02 17:42:37.073672 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:37.073629 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:42:37.104138 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:37.104100 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7df526\"/\"distinct-2-w6-7df526-kserve-self-signed-certs\"" Nov 02 17:42:37.106907 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:37.106885 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j"] Nov 02 17:42:37.200347 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:37.200274 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-dshm\") pod \"distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j\" (UID: \"3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:42:37.200524 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:37.200430 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-home\") pod \"distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j\" (UID: \"3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:42:37.200524 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:37.200468 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-tls-certs\") pod \"distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j\" (UID: \"3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:42:37.200600 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:37.200564 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-tmp-dir\") pod \"distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j\" (UID: \"3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:42:37.200600 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:37.200584 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjpxh\" (UniqueName: \"kubernetes.io/projected/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-kube-api-access-rjpxh\") pod \"distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j\" (UID: \"3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:42:37.200734 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:37.200709 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-model-cache\") pod \"distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j\" (UID: \"3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:42:37.314153 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:37.314108 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-dshm\") pod \"distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j\" (UID: \"3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:42:37.314971 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:37.314945 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-home\") pod \"distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j\" (UID: \"3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:42:37.315592 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:37.315572 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-tls-certs\") pod \"distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j\" (UID: \"3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:42:37.342182 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:37.342105 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-tmp-dir\") pod \"distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j\" (UID: \"3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:42:37.342810 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:37.342787 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjpxh\" (UniqueName: \"kubernetes.io/projected/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-kube-api-access-rjpxh\") pod \"distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j\" (UID: \"3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:42:37.343377 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:37.343353 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-model-cache\") pod \"distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j\" (UID: \"3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:42:37.368327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:37.315489 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-home\") pod \"distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j\" (UID: \"3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:42:37.368614 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:37.342719 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-tmp-dir\") pod \"distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j\" (UID: \"3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:42:37.368757 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:37.343739 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-model-cache\") pod \"distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j\" (UID: \"3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:42:37.368864 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:37.351739 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-tls-certs\") pod \"distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j\" (UID: \"3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:42:37.368968 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:37.368938 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjpxh\" (UniqueName: \"kubernetes.io/projected/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-kube-api-access-rjpxh\") pod \"distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j\" (UID: \"3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:42:37.369140 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:37.364980 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-dshm\") pod \"distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j\" (UID: \"3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:42:37.386591 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:37.386558 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:42:37.692411 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:42:37.692376 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c39cbb8_3b30_4f0e_b0f3_4685433cc2d5.slice/crio-ff26e7fc28dde8e4228695dfcf4490c9df228f9b373fce4a9c70f96ff098bd94 WatchSource:0}: Error finding container ff26e7fc28dde8e4228695dfcf4490c9df228f9b373fce4a9c70f96ff098bd94: Status 404 returned error can't find the container with id ff26e7fc28dde8e4228695dfcf4490c9df228f9b373fce4a9c70f96ff098bd94 Nov 02 17:42:37.693334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:37.693286 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j"] Nov 02 17:42:37.757843 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:37.757810 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" event={"ID":"3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5","Type":"ContainerStarted","Data":"2306a65cf2a35c75f09a07854c5c8050020be427883c4c8dc5943663abe6ad84"} Nov 02 17:42:37.757980 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:37.757863 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" event={"ID":"3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5","Type":"ContainerStarted","Data":"ff26e7fc28dde8e4228695dfcf4490c9df228f9b373fce4a9c70f96ff098bd94"} Nov 02 17:42:37.802191 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:37.802140 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podStartSLOduration=0.802125955 podStartE2EDuration="802.125955ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:37.801453254 +0000 UTC m=+775.496808740" watchObservedRunningTime="2026-11-02 17:42:37.802125955 +0000 UTC m=+775.497481429" Nov 02 17:42:38.388651 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:38.388615 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:42:39.184459 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:42:39.184423 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:42:46.301808 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.301771 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r"] Nov 02 17:42:46.306371 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.306344 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:42:46.315355 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.315329 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4"] Nov 02 17:42:46.319407 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.319384 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:42:46.333819 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.333785 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m"] Nov 02 17:42:46.337957 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.337932 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:42:46.348073 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.346360 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r"] Nov 02 17:42:46.350163 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.350133 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4"] Nov 02 17:42:46.368148 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.368112 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-credential-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:42:46.368293 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.368168 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istio-podinfo\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:42:46.368293 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.368205 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istio-token\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:42:46.368293 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.368252 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istio-data\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:42:46.368602 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.368566 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istiod-ca-cert\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:42:46.368675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.368649 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istiod-ca-cert\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:42:46.368740 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.368715 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-credential-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:42:46.368826 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.368754 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istio-envoy\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:42:46.368826 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.368807 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istio-envoy\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:42:46.368902 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.368846 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istio-token\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:42:46.368902 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.368871 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ntzb\" (UniqueName: \"kubernetes.io/projected/ce398a00-0c4b-45ea-a920-acb66ae85b7f-kube-api-access-6ntzb\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:42:46.368987 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.368905 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-workload-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:42:46.368987 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.368965 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62smk\" (UniqueName: \"kubernetes.io/projected/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-kube-api-access-62smk\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:42:46.369058 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.368988 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-workload-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:42:46.369058 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.369004 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-workload-certs\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:42:46.369133 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.369070 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-workload-certs\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:42:46.369133 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.369088 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istio-podinfo\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:42:46.369133 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.369118 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istio-data\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:42:46.402415 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.402379 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv"] Nov 02 17:42:46.414391 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.413969 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv"] Nov 02 17:42:46.415291 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.414948 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:42:46.430358 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.426040 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m"] Nov 02 17:42:46.430516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.430425 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv"] Nov 02 17:42:46.430516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.426130 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:42:46.444848 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.444816 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv"] Nov 02 17:42:46.470884 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.470847 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-workload-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:42:46.471055 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.471032 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0316e437-096a-4330-b71c-3b33b139c4d9-istio-podinfo\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:42:46.471118 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.471075 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istio-token\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:42:46.471180 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.471148 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-credential-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:42:46.471235 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.471186 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istio-podinfo\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:42:46.471235 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.471205 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-workload-certs\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:42:46.471235 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.471231 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istio-token\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:42:46.471407 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.471246 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ca3f757-188e-4202-9995-fd167353d6be-istio-token\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:42:46.471407 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.471391 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-workload-certs\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:42:46.471512 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.471408 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istio-data\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:42:46.471512 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.471437 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istio-data\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:42:46.471512 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.471461 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-workload-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:42:46.471512 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.471505 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kls7\" (UniqueName: \"kubernetes.io/projected/0316e437-096a-4330-b71c-3b33b139c4d9-kube-api-access-8kls7\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:42:46.471708 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.471552 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-credential-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:42:46.471708 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.471591 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istiod-ca-cert\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:42:46.471708 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.471645 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istiod-ca-cert\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:42:46.471708 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.471669 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ca3f757-188e-4202-9995-fd167353d6be-istio-podinfo\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:42:46.471881 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.471807 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istio-data\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:42:46.471881 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.471811 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ca3f757-188e-4202-9995-fd167353d6be-istiod-ca-cert\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:42:46.471991 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.471894 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-credential-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:42:46.471991 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.471939 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istio-envoy\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:42:46.472095 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.471991 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istio-envoy\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:42:46.472095 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.472023 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffr74\" (UniqueName: \"kubernetes.io/projected/7d2e5cde-97d4-4e6a-a564-55693ade99ab-kube-api-access-ffr74\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:42:46.472095 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.472067 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0316e437-096a-4330-b71c-3b33b139c4d9-istio-token\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:42:46.472277 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.472093 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istio-token\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:42:46.472277 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.472121 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-workload-certs\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:42:46.472277 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.472158 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ntzb\" (UniqueName: \"kubernetes.io/projected/ce398a00-0c4b-45ea-a920-acb66ae85b7f-kube-api-access-6ntzb\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:42:46.472277 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.472170 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-credential-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:42:46.472572 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.472283 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istiod-ca-cert\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:42:46.472572 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.472354 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-istio-envoy\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:42:46.472572 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.472355 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istiod-ca-cert\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:42:46.472572 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.472401 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrccr\" (UniqueName: \"kubernetes.io/projected/1ca3f757-188e-4202-9995-fd167353d6be-kube-api-access-qrccr\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:42:46.472572 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.472462 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-istio-data\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:42:46.472572 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.472508 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-istio-data\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:42:46.472808 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.472621 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-workload-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:42:46.472808 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.472653 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62smk\" (UniqueName: \"kubernetes.io/projected/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-kube-api-access-62smk\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:42:46.472808 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.472678 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-workload-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:42:46.472808 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.472704 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-workload-certs\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:42:46.472808 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.472748 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0316e437-096a-4330-b71c-3b33b139c4d9-istiod-ca-cert\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:42:46.473121 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.472843 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-istio-envoy\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:42:46.473121 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.472994 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-workload-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:42:46.473121 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.473096 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-workload-certs\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:42:46.473273 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.473180 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-workload-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:42:46.473273 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.473185 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istiod-ca-cert\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:42:46.473402 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.473280 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-workload-certs\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:42:46.473402 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.473324 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istio-podinfo\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:42:46.473402 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.473357 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-credential-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:42:46.473402 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.473391 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istio-podinfo\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:42:46.473612 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.473428 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-credential-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:42:46.473612 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.473451 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istio-envoy\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:42:46.473612 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.473510 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istio-data\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:42:46.473612 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.473545 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-credential-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:42:46.473821 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.473766 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-workload-certs\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:42:46.473887 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.473847 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istio-data\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:42:46.473970 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.473944 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-workload-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:42:46.474437 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.474413 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istio-envoy\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:42:46.474556 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.474536 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istio-envoy\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:42:46.474602 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.474553 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istio-podinfo\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:42:46.475866 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.475844 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istio-podinfo\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:42:46.498337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.498294 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istio-token\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:42:46.498518 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.498490 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istio-token\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:42:46.499015 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.498989 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ntzb\" (UniqueName: \"kubernetes.io/projected/ce398a00-0c4b-45ea-a920-acb66ae85b7f-kube-api-access-6ntzb\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:42:46.499236 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.499214 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62smk\" (UniqueName: \"kubernetes.io/projected/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-kube-api-access-62smk\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:42:46.575636 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.575559 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istio-podinfo\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:42:46.575636 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.575595 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-credential-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:42:46.575636 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.575611 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istio-envoy\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:42:46.575874 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.575638 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-credential-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:42:46.575874 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.575669 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-workload-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:42:46.575874 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.575741 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-workload-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:42:46.575874 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.575790 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0316e437-096a-4330-b71c-3b33b139c4d9-istio-podinfo\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:42:46.575874 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.575815 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istio-token\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:42:46.575874 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.575852 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-workload-certs\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:42:46.576193 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.575893 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ca3f757-188e-4202-9995-fd167353d6be-istio-token\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:42:46.576193 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.575920 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-workload-certs\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:42:46.576193 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.575944 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istio-data\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:42:46.576193 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.575987 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-workload-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:42:46.576193 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.576010 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kls7\" (UniqueName: \"kubernetes.io/projected/0316e437-096a-4330-b71c-3b33b139c4d9-kube-api-access-8kls7\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:42:46.576193 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.576081 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-credential-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:42:46.576193 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.576086 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ca3f757-188e-4202-9995-fd167353d6be-istio-podinfo\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:42:46.576193 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.576114 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-credential-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:42:46.576193 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.576132 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ca3f757-188e-4202-9995-fd167353d6be-istiod-ca-cert\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:42:46.576193 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.576163 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-workload-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:42:46.577404 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.576206 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-workload-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:42:46.577404 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.576244 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffr74\" (UniqueName: \"kubernetes.io/projected/7d2e5cde-97d4-4e6a-a564-55693ade99ab-kube-api-access-ffr74\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:42:46.577404 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.576293 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0316e437-096a-4330-b71c-3b33b139c4d9-istio-token\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:42:46.577404 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.576354 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-workload-certs\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:42:46.577404 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.576408 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-istio-envoy\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:42:46.577404 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.576435 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrccr\" (UniqueName: \"kubernetes.io/projected/1ca3f757-188e-4202-9995-fd167353d6be-kube-api-access-qrccr\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:42:46.577404 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.576461 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-istio-data\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:42:46.577404 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.576487 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-istio-data\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:42:46.577404 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.576533 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0316e437-096a-4330-b71c-3b33b139c4d9-istiod-ca-cert\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:42:46.577404 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.576559 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-istio-envoy\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:42:46.577404 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.576588 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istiod-ca-cert\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:42:46.577404 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.576628 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-credential-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:42:46.577404 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.576783 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-workload-certs\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:42:46.577404 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.576865 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-workload-certs\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:42:46.577404 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.576890 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ca3f757-188e-4202-9995-fd167353d6be-istiod-ca-cert\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:42:46.577404 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.576977 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-credential-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:42:46.577404 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.577078 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-workload-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:42:46.577404 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.577203 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-istio-data\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:42:46.577404 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.577229 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-istio-data\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:42:46.577404 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.577368 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-workload-certs\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:42:46.578448 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.577883 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0316e437-096a-4330-b71c-3b33b139c4d9-istiod-ca-cert\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:42:46.578448 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.576532 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istio-data\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:42:46.578448 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.578261 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istiod-ca-cert\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:42:46.578696 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.578676 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istio-envoy\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:42:46.578983 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.578958 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ca3f757-188e-4202-9995-fd167353d6be-istio-podinfo\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:42:46.579247 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.579224 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istio-podinfo\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:42:46.579409 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.579391 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-istio-envoy\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:42:46.579707 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.579690 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-istio-envoy\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:42:46.580384 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.580350 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0316e437-096a-4330-b71c-3b33b139c4d9-istio-podinfo\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:42:46.600661 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.600636 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istio-token\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:42:46.600771 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.600660 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ca3f757-188e-4202-9995-fd167353d6be-istio-token\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:42:46.600976 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.600951 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0316e437-096a-4330-b71c-3b33b139c4d9-istio-token\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:42:46.601330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.601285 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kls7\" (UniqueName: \"kubernetes.io/projected/0316e437-096a-4330-b71c-3b33b139c4d9-kube-api-access-8kls7\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:42:46.601485 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.601471 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrccr\" (UniqueName: \"kubernetes.io/projected/1ca3f757-188e-4202-9995-fd167353d6be-kube-api-access-qrccr\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:42:46.601690 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.601669 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffr74\" (UniqueName: \"kubernetes.io/projected/7d2e5cde-97d4-4e6a-a564-55693ade99ab-kube-api-access-ffr74\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:42:46.622516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.622497 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:42:46.633497 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.633477 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:42:46.659970 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.659940 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:42:46.748091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.748054 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:42:46.758596 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.758571 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:42:46.941733 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:42:46.941690 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a1e0e8_1ff4_4ba9_8b2e_fef4cb120afd.slice/crio-c8415ffafdc2306e5913b2bc8c6ab33959e65dabf86930b78c7bd23c533b7f08 WatchSource:0}: Error finding container c8415ffafdc2306e5913b2bc8c6ab33959e65dabf86930b78c7bd23c533b7f08: Status 404 returned error can't find the container with id c8415ffafdc2306e5913b2bc8c6ab33959e65dabf86930b78c7bd23c533b7f08 Nov 02 17:42:46.943319 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.943288 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r"] Nov 02 17:42:46.944439 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.944411 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:42:46.944515 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.944472 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:42:46.944515 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.944502 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:42:46.955508 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:42:46.955481 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce398a00_0c4b_45ea_a920_acb66ae85b7f.slice/crio-c1ac45ad6504c66f0442ccdc77f7e594533d1797d0f4164d1d60bb6a05f4afb4 WatchSource:0}: Error finding container c1ac45ad6504c66f0442ccdc77f7e594533d1797d0f4164d1d60bb6a05f4afb4: Status 404 returned error can't find the container with id c1ac45ad6504c66f0442ccdc77f7e594533d1797d0f4164d1d60bb6a05f4afb4 Nov 02 17:42:46.957270 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.957239 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4"] Nov 02 17:42:46.958400 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.958369 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:42:46.958467 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.958440 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:42:46.958511 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:46.958471 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:42:47.065872 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:47.064894 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m"] Nov 02 17:42:47.068089 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:42:47.068072 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0316e437_096a_4330_b71c_3b33b139c4d9.slice/crio-fcf7385ea1cc7c3e8c89b9fd3a47499ea0eb8f00540957dbb1dcee70a0ae1779 WatchSource:0}: Error finding container fcf7385ea1cc7c3e8c89b9fd3a47499ea0eb8f00540957dbb1dcee70a0ae1779: Status 404 returned error can't find the container with id fcf7385ea1cc7c3e8c89b9fd3a47499ea0eb8f00540957dbb1dcee70a0ae1779 Nov 02 17:42:47.071178 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:47.071148 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:42:47.071275 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:47.071212 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:42:47.071275 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:47.071245 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:42:47.104950 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:47.104653 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv"] Nov 02 17:42:47.108055 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:47.107832 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv"] Nov 02 17:42:47.119233 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:47.118617 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:42:47.119233 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:47.118742 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:42:47.119233 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:47.118800 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:42:47.148528 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:47.142472 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:42:47.148528 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:47.142588 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:42:47.148528 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:47.142649 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:42:47.396381 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:47.395731 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:42:47.432288 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:47.432003 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:42:47.826286 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:47.826250 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" event={"ID":"ce398a00-0c4b-45ea-a920-acb66ae85b7f","Type":"ContainerStarted","Data":"db4079d9e3d63af66da8c3b85be9318801464214ff32c4ab22c99fe95aa6b900"} Nov 02 17:42:47.826497 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:47.826324 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" event={"ID":"ce398a00-0c4b-45ea-a920-acb66ae85b7f","Type":"ContainerStarted","Data":"c1ac45ad6504c66f0442ccdc77f7e594533d1797d0f4164d1d60bb6a05f4afb4"} Nov 02 17:42:47.828402 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:47.828376 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" event={"ID":"1ca3f757-188e-4202-9995-fd167353d6be","Type":"ContainerStarted","Data":"9fa24b625137ca3eabcc24a8a897062c12ffe0d82e560295f2147e7c8fcc9029"} Nov 02 17:42:47.828546 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:47.828417 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" event={"ID":"1ca3f757-188e-4202-9995-fd167353d6be","Type":"ContainerStarted","Data":"3a43a627b379567403fde8adbd0a70166c0cb3a97e4bbbe4ebc4c8b64bb8488b"} Nov 02 17:42:47.829989 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:47.829959 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" event={"ID":"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd","Type":"ContainerStarted","Data":"2d7f54c0b0acf24b5edbaa375a560d5d2cc71593c98d66c99eccca24b7b9dbdc"} Nov 02 17:42:47.830091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:47.830005 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" event={"ID":"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd","Type":"ContainerStarted","Data":"c8415ffafdc2306e5913b2bc8c6ab33959e65dabf86930b78c7bd23c533b7f08"} Nov 02 17:42:47.831483 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:47.831459 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" event={"ID":"7d2e5cde-97d4-4e6a-a564-55693ade99ab","Type":"ContainerStarted","Data":"f2064eb83d67fba7b2369044bbd4d1e37ee55d5e969026d4d188857c5d6d053b"} Nov 02 17:42:47.831597 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:47.831507 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" event={"ID":"7d2e5cde-97d4-4e6a-a564-55693ade99ab","Type":"ContainerStarted","Data":"066a19770511dab549ccb96fc95fe9ad10522dd698f80ec45f44a25b92811f01"} Nov 02 17:42:47.833054 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:47.833030 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" event={"ID":"0316e437-096a-4330-b71c-3b33b139c4d9","Type":"ContainerStarted","Data":"e9e0d421d895df66f634f9cc802c2666a0495c29040e2d6f268e87b5da3d400b"} Nov 02 17:42:47.833138 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:47.833075 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" event={"ID":"0316e437-096a-4330-b71c-3b33b139c4d9","Type":"ContainerStarted","Data":"fcf7385ea1cc7c3e8c89b9fd3a47499ea0eb8f00540957dbb1dcee70a0ae1779"} Nov 02 17:42:47.844900 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:47.844882 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:42:47.877237 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:47.877195 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" podStartSLOduration=1.877182666 podStartE2EDuration="1.877182666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:47.874856673 +0000 UTC m=+785.570212148" watchObservedRunningTime="2026-11-02 17:42:47.877182666 +0000 UTC m=+785.572538139" Nov 02 17:42:47.980890 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:47.980811 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" podStartSLOduration=1.980788014 podStartE2EDuration="1.980788014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:47.977672084 +0000 UTC m=+785.673027583" watchObservedRunningTime="2026-11-02 17:42:47.980788014 +0000 UTC m=+785.676143488" Nov 02 17:42:48.058012 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:48.052548 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" podStartSLOduration=2.052524079 podStartE2EDuration="2.052524079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:48.050847246 +0000 UTC m=+785.746202751" watchObservedRunningTime="2026-11-02 17:42:48.052524079 +0000 UTC m=+785.747879566" Nov 02 17:42:48.213406 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:48.212911 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" podStartSLOduration=2.212891988 podStartE2EDuration="2.212891988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:48.152657446 +0000 UTC m=+785.848012970" watchObservedRunningTime="2026-11-02 17:42:48.212891988 +0000 UTC m=+785.908247461" Nov 02 17:42:48.235330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:48.230954 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" podStartSLOduration=2.230936516 podStartE2EDuration="2.230936516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:48.210514803 +0000 UTC m=+785.905870303" watchObservedRunningTime="2026-11-02 17:42:48.230936516 +0000 UTC m=+785.926291990" Nov 02 17:42:48.706332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:48.687069 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 17:42:48.706332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:48.687230 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" podUID="0316e437-096a-4330-b71c-3b33b139c4d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 17:42:48.706332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:48.687122 4471 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 02 17:42:48.706332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:48.687493 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" podUID="c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 02 17:42:48.706332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:48.702489 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:42:48.706332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:48.702570 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" podUID="ce398a00-0c4b-45ea-a920-acb66ae85b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 17:42:48.778333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:48.746691 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:42:48.826491 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:48.826449 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:42:48.826799 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:48.826779 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:42:48.826941 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:48.826930 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:42:48.827065 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:48.827054 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:42:48.827185 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:48.771182 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 17:42:48.827367 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:48.827342 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" podUID="7d2e5cde-97d4-4e6a-a564-55693ade99ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 17:42:48.827542 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:48.814464 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 17:42:48.827665 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:48.827646 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" podUID="1ca3f757-188e-4202-9995-fd167353d6be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 17:42:49.260279 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:49.260229 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 02 17:42:49.299467 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:49.299421 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:49.309948 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:49.309899 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:49.473512 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:49.473461 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:49.656331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:49.643709 4471 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 02 17:42:49.656331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:49.643770 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" podUID="c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 02 17:42:49.656331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:49.644458 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:42:49.656331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:49.644504 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" podUID="ce398a00-0c4b-45ea-a920-acb66ae85b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 17:42:49.678001 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:49.677958 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:49.698526 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:49.698485 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 17:42:49.698849 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:49.698817 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" podUID="0316e437-096a-4330-b71c-3b33b139c4d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 17:42:49.751134 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:49.751096 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 17:42:49.751766 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:49.751733 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" podUID="7d2e5cde-97d4-4e6a-a564-55693ade99ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 17:42:49.760749 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:49.760716 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 17:42:49.760988 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:49.760958 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" podUID="1ca3f757-188e-4202-9995-fd167353d6be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 17:42:49.841962 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:49.841899 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:50.139444 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:50.139382 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 02 17:42:50.276329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:50.265391 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:50.276329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:50.266043 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:42:50.312336 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:50.308501 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:50.312336 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:50.309238 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 17:42:50.312336 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:50.310392 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:50.312336 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:50.310567 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:42:50.582331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:50.566420 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:42:50.662330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:50.661719 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:42:50.662330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:50.661784 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" podUID="ce398a00-0c4b-45ea-a920-acb66ae85b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 17:42:50.711332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:50.662644 4471 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 02 17:42:50.711332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:50.662691 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" podUID="c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 02 17:42:50.711332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:50.662766 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 17:42:50.711332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:50.662789 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" podUID="0316e437-096a-4330-b71c-3b33b139c4d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 17:42:50.771334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:50.762966 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 17:42:50.771334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:50.763026 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" podUID="7d2e5cde-97d4-4e6a-a564-55693ade99ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 17:42:50.771334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:50.763401 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 17:42:50.771334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:50.763442 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" podUID="1ca3f757-188e-4202-9995-fd167353d6be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 17:42:50.876446 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:50.843048 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 02 17:42:50.959341 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:50.919002 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:42:50.959341 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:50.919068 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:42:51.147539 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:51.147431 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:51.360893 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:51.298331 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:42:51.614332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:51.332999 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": context deadline exceeded" Nov 02 17:42:51.614332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:51.340055 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:51.614332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:51.378780 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 02 17:42:51.627329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:51.588843 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:51.677331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:51.631120 4471 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 02 17:42:51.677331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:51.631168 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" podUID="c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 02 17:42:51.677331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:51.642634 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:42:51.677331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:51.642683 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" podUID="ce398a00-0c4b-45ea-a920-acb66ae85b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 17:42:51.677697 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:51.677521 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:42:51.701330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:51.694404 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:51.724459 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:51.724422 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 17:42:51.725519 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:51.725485 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" podUID="0316e437-096a-4330-b71c-3b33b139c4d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 17:42:51.779330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:51.769665 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 17:42:51.779330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:51.769724 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" podUID="7d2e5cde-97d4-4e6a-a564-55693ade99ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 17:42:51.779330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:51.776901 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 17:42:51.779330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:51.776948 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" podUID="1ca3f757-188e-4202-9995-fd167353d6be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 17:42:51.848328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:51.847927 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:52.306334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:52.290494 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 02 17:42:52.627833 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:52.627731 4471 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 02 17:42:52.628073 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:52.628042 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" podUID="c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 02 17:42:52.629443 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:52.629405 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:52.630968 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:52.630941 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": context deadline exceeded" Nov 02 17:42:52.631426 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:52.631398 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 02 17:42:52.677354 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:52.642210 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:42:52.677354 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:52.642273 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" podUID="ce398a00-0c4b-45ea-a920-acb66ae85b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 17:42:52.677354 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:52.662460 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 17:42:52.677354 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:52.662513 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" podUID="0316e437-096a-4330-b71c-3b33b139c4d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 17:42:52.850328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:52.849860 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 02 17:42:52.879382 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:52.876129 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 17:42:52.929158 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:52.929115 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" podUID="1ca3f757-188e-4202-9995-fd167353d6be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 17:42:52.929430 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:52.883375 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:52.929967 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:52.879515 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 17:42:52.941567 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:52.941529 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" podUID="7d2e5cde-97d4-4e6a-a564-55693ade99ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 17:42:52.941775 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:52.931920 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:42:53.366326 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:53.359441 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:53.463342 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:53.462771 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:42:53.627541 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:53.627439 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:53.629803 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:53.629772 4471 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 02 17:42:53.632408 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:53.632375 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" podUID="c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 02 17:42:53.636950 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:53.636925 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:42:53.637117 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:53.637093 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" podUID="ce398a00-0c4b-45ea-a920-acb66ae85b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 17:42:53.637321 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:53.637287 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:53.637714 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:53.637688 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:53.639403 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:53.638037 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": context deadline exceeded" Nov 02 17:42:53.663760 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:53.663726 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 17:42:53.664422 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:53.664388 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" podUID="0316e437-096a-4330-b71c-3b33b139c4d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 17:42:53.775042 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:53.775006 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 17:42:53.776845 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:53.776814 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" podUID="7d2e5cde-97d4-4e6a-a564-55693ade99ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 17:42:53.777028 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:53.775333 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 17:42:53.777136 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:53.777117 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" podUID="1ca3f757-188e-4202-9995-fd167353d6be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 17:42:53.858332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:53.857226 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:53.945390 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:53.941828 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:54.369340 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:54.365410 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:54.629718 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:54.629602 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 02 17:42:54.630749 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:54.630728 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:42:54.648190 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:54.648142 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:54.666358 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:54.648506 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:42:54.668517 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:54.648536 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 02 17:42:54.673325 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:54.671275 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:42:54.673325 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:54.671344 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" podUID="ce398a00-0c4b-45ea-a920-acb66ae85b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 17:42:54.673325 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:54.672159 4471 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 02 17:42:54.673325 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:54.672195 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" podUID="c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 02 17:42:54.703338 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:54.674793 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 17:42:54.716232 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:54.716192 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" podUID="0316e437-096a-4330-b71c-3b33b139c4d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 17:42:54.753963 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:54.753914 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 17:42:54.754272 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:54.754225 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" podUID="7d2e5cde-97d4-4e6a-a564-55693ade99ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 17:42:54.762853 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:54.762730 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 17:42:54.762853 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:54.762785 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" podUID="1ca3f757-188e-4202-9995-fd167353d6be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 17:42:54.861236 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:54.860419 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 02 17:42:54.928994 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:54.927413 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:42:54.970406 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:54.945284 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:55.385283 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:55.370616 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:55.631352 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:55.630942 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:55.677864 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:55.677763 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 02 17:42:55.711332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:55.686132 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": context deadline exceeded" Nov 02 17:42:55.711941 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:55.695489 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:42:55.724332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:55.695627 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:42:55.724621 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:55.724583 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" podUID="ce398a00-0c4b-45ea-a920-acb66ae85b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 17:42:55.763330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:55.738160 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 17:42:55.763644 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:55.763602 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" podUID="0316e437-096a-4330-b71c-3b33b139c4d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 17:42:55.763805 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:55.756638 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 17:42:55.763939 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:55.763913 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" podUID="7d2e5cde-97d4-4e6a-a564-55693ade99ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 17:42:55.783216 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:55.782582 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 17:42:55.783216 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:55.782646 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" podUID="1ca3f757-188e-4202-9995-fd167353d6be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 17:42:55.845073 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:55.844938 4471 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 02 17:42:55.845073 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:55.844998 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" podUID="c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 02 17:42:55.864163 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:55.862400 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:55.928807 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:55.926976 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 17:42:56.386331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:56.381803 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:56.631328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:56.630788 4471 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 02 17:42:56.631328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:56.630868 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" podUID="c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 02 17:42:56.632662 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:56.631592 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 02 17:42:56.646391 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:56.645434 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:42:56.646391 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:56.645541 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" podUID="ce398a00-0c4b-45ea-a920-acb66ae85b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 17:42:56.671331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:56.670072 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 17:42:56.671331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:56.670134 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" podUID="0316e437-096a-4330-b71c-3b33b139c4d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 17:42:56.687298 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:56.686431 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:56.712496 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:56.712452 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:56.713191 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:56.712728 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": context deadline exceeded" Nov 02 17:42:56.766672 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:56.765999 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 17:42:56.766672 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:56.766063 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" podUID="1ca3f757-188e-4202-9995-fd167353d6be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 17:42:56.767047 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:56.767026 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 17:42:56.767185 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:56.767164 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" podUID="7d2e5cde-97d4-4e6a-a564-55693ade99ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 17:42:56.868330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:56.867450 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:56.942469 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:56.938201 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:56.968283 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:56.968251 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:42:57.396374 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:57.380985 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": context deadline exceeded" Nov 02 17:42:57.442359 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:57.431133 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:57.517351 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:57.514612 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:57.663840 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:57.663398 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 02 17:42:57.668600 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:57.668441 4471 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 02 17:42:57.668600 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:57.668519 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" podUID="c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 02 17:42:57.670029 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:57.669173 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 17:42:57.670029 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:57.669224 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" podUID="0316e437-096a-4330-b71c-3b33b139c4d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 17:42:57.691169 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:57.691129 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:42:57.700073 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:57.700021 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" podUID="ce398a00-0c4b-45ea-a920-acb66ae85b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 17:42:57.700266 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:57.692905 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:57.732332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:57.726460 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:42:57.732332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:57.727805 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:57.749168 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:57.749122 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 17:42:57.749392 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:57.749202 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" podUID="7d2e5cde-97d4-4e6a-a564-55693ade99ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 17:42:57.842330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:57.841801 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 17:42:57.842330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:57.841873 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" podUID="1ca3f757-188e-4202-9995-fd167353d6be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 17:42:57.890468 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:57.890416 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 02 17:42:58.002775 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:58.002681 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:58.109501 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:58.109446 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:58.447327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:58.442602 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 02 17:42:58.461296 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:58.461252 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:58.628746 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:58.628707 4471 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 02 17:42:58.629021 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:58.628987 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" podUID="c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 02 17:42:58.635958 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:58.635925 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:42:58.636196 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:58.636171 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" podUID="ce398a00-0c4b-45ea-a920-acb66ae85b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 17:42:58.661245 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:58.661212 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 17:42:58.661555 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:58.661510 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" podUID="0316e437-096a-4330-b71c-3b33b139c4d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 17:42:58.674449 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:58.674411 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:58.684442 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:58.684409 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:42:58.708385 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:58.702438 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:58.712482 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:58.712445 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:58.727727 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:58.727687 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:58.728608 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:58.728579 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": context deadline exceeded" Nov 02 17:42:58.772937 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:58.772895 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 17:42:58.776564 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:58.776527 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" podUID="1ca3f757-188e-4202-9995-fd167353d6be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 17:42:58.777758 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:58.777737 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:42:58.949017 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:58.948968 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:42:58.973689 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:58.954902 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:42:58.998110 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:58.998079 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:42:59.039918 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:59.039887 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:42:59.066399 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:59.066359 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 17:42:59.066672 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:59.066631 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" podUID="7d2e5cde-97d4-4e6a-a564-55693ade99ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 17:42:59.223566 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:59.223234 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 17:42:59.225377 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:59.224824 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:42:59.715330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:59.706352 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:42:59.715330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:59.708059 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:42:59.726330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:42:59.725228 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:43:00.002784 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:00.002687 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:43:00.004175 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:00.004150 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:43:00.006621 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:00.006600 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:43:00.058829 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:00.058796 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:43:00.059759 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:00.059727 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:43:00.060457 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:00.060423 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:43:00.129673 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:00.129641 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:43:00.130291 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:00.130254 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:43:00.131498 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:00.131476 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:43:00.185340 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:00.185271 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:43:00.187007 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:00.186985 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:43:00.188383 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:00.188354 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:43:09.295774 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:43:09.295730 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:43:16.462789 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:16.462753 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2"] Nov 02 17:43:16.467206 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:16.467185 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 17:43:16.505060 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:16.505028 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2"] Nov 02 17:43:16.614719 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:16.614686 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-istio-data\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 17:43:16.614719 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:16.614730 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81708d59-e82f-4259-9bf1-cec029c05c73-istio-token\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 17:43:16.614945 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:16.614816 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81708d59-e82f-4259-9bf1-cec029c05c73-istiod-ca-cert\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 17:43:16.614945 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:16.614907 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sdj5\" (UniqueName: \"kubernetes.io/projected/81708d59-e82f-4259-9bf1-cec029c05c73-kube-api-access-6sdj5\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 17:43:16.615075 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:16.615039 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81708d59-e82f-4259-9bf1-cec029c05c73-istio-podinfo\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 17:43:16.615136 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:16.615116 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-credential-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 17:43:16.615194 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:16.615157 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-workload-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 17:43:16.615194 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:16.615182 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-istio-envoy\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 17:43:16.615605 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:16.615416 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-workload-certs\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 17:43:16.716966 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:16.716893 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81708d59-e82f-4259-9bf1-cec029c05c73-istio-podinfo\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 17:43:16.716966 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:16.716936 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-credential-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 17:43:16.716966 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:16.716958 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-workload-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 17:43:16.717181 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:16.716975 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-istio-envoy\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 17:43:16.717181 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:16.717075 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-workload-certs\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 17:43:16.717270 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:16.717202 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-istio-data\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 17:43:16.717270 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:16.717234 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81708d59-e82f-4259-9bf1-cec029c05c73-istio-token\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 17:43:16.717270 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:16.717266 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81708d59-e82f-4259-9bf1-cec029c05c73-istiod-ca-cert\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 17:43:16.717451 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:16.717346 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sdj5\" (UniqueName: \"kubernetes.io/projected/81708d59-e82f-4259-9bf1-cec029c05c73-kube-api-access-6sdj5\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 17:43:16.717451 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:16.717362 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-credential-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 17:43:16.717451 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:16.717403 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-workload-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 17:43:16.717610 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:16.717504 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-workload-certs\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 17:43:16.717863 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:16.717841 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-istio-data\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 17:43:16.718161 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:16.718138 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81708d59-e82f-4259-9bf1-cec029c05c73-istiod-ca-cert\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 17:43:16.719413 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:16.719394 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-istio-envoy\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 17:43:16.719663 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:16.719646 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81708d59-e82f-4259-9bf1-cec029c05c73-istio-podinfo\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 17:43:16.744173 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:16.744150 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81708d59-e82f-4259-9bf1-cec029c05c73-istio-token\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 17:43:16.745498 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:16.745479 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sdj5\" (UniqueName: \"kubernetes.io/projected/81708d59-e82f-4259-9bf1-cec029c05c73-kube-api-access-6sdj5\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 17:43:16.793670 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:16.793647 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 17:43:17.112987 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:43:17.112959 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81708d59_e82f_4259_9bf1_cec029c05c73.slice/crio-6777cc5d2ffa9d099ad40ff20a5234949cdd1153c04eca35f36dc548c4c4062c WatchSource:0}: Error finding container 6777cc5d2ffa9d099ad40ff20a5234949cdd1153c04eca35f36dc548c4c4062c: Status 404 returned error can't find the container with id 6777cc5d2ffa9d099ad40ff20a5234949cdd1153c04eca35f36dc548c4c4062c Nov 02 17:43:17.114760 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:17.114737 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2"] Nov 02 17:43:17.115473 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:17.115443 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:43:17.115546 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:17.115507 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:43:17.115546 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:17.115537 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:43:18.108021 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:18.107982 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" event={"ID":"81708d59-e82f-4259-9bf1-cec029c05c73","Type":"ContainerStarted","Data":"9d0c192800201632bd49dc7cdd0360bdce4d1074c206f02d30b64d3baf2c16a9"} Nov 02 17:43:18.108431 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:18.108063 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" event={"ID":"81708d59-e82f-4259-9bf1-cec029c05c73","Type":"ContainerStarted","Data":"6777cc5d2ffa9d099ad40ff20a5234949cdd1153c04eca35f36dc548c4c4062c"} Nov 02 17:43:18.180259 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:18.180195 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" podStartSLOduration=2.180176508 podStartE2EDuration="2.180176508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:18.179620481 +0000 UTC m=+815.874975989" watchObservedRunningTime="2026-11-02 17:43:18.180176508 +0000 UTC m=+815.875531982" Nov 02 17:43:18.793825 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:18.793792 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 17:43:18.795166 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:18.795135 4471 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 17:43:18.795333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:18.795195 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" podUID="81708d59-e82f-4259-9bf1-cec029c05c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 17:43:19.092697 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:19.092607 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5"] Nov 02 17:43:19.093156 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:19.093092 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" podUID="7689ac62-6665-4a39-80f2-1a3785898881" containerName="istio-proxy" containerID="cri-o://c556e00298ef95e9632c853cda6e55e2bf0bc1f340f88982b09be25ef1441379" gracePeriod=30 Nov 02 17:43:19.794969 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:19.794926 4471 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 17:43:19.795502 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:19.794997 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" podUID="81708d59-e82f-4259-9bf1-cec029c05c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 17:43:20.794754 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:20.794720 4471 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 17:43:20.794927 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:20.794774 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" podUID="81708d59-e82f-4259-9bf1-cec029c05c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 17:43:21.794549 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:21.794508 4471 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 17:43:21.794966 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:21.794583 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" podUID="81708d59-e82f-4259-9bf1-cec029c05c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 17:43:22.794426 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:22.794388 4471 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 17:43:22.794627 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:22.794464 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" podUID="81708d59-e82f-4259-9bf1-cec029c05c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 17:43:23.794495 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:23.794456 4471 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 17:43:23.794676 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:23.794534 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" podUID="81708d59-e82f-4259-9bf1-cec029c05c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 17:43:24.363141 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:24.363079 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:43:24.461318 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:24.461269 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-istio-data\") pod \"7689ac62-6665-4a39-80f2-1a3785898881\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " Nov 02 17:43:24.461505 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:24.461341 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-credential-socket\") pod \"7689ac62-6665-4a39-80f2-1a3785898881\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " Nov 02 17:43:24.461505 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:24.461418 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-workload-socket\") pod \"7689ac62-6665-4a39-80f2-1a3785898881\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " Nov 02 17:43:24.461643 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:24.461564 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-istio-data" pod "7689ac62-6665-4a39-80f2-1a3785898881" (UID: "7689ac62-6665-4a39-80f2-1a3785898881"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:43:24.461643 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:24.461583 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7689ac62-6665-4a39-80f2-1a3785898881-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7689ac62-6665-4a39-80f2-1a3785898881-istio-podinfo\") pod \"7689ac62-6665-4a39-80f2-1a3785898881\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " Nov 02 17:43:24.461750 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:24.461657 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-credential-socket" pod "7689ac62-6665-4a39-80f2-1a3785898881" (UID: "7689ac62-6665-4a39-80f2-1a3785898881"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:43:24.461750 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:24.461674 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7689ac62-6665-4a39-80f2-1a3785898881-kube-api-access-f7jqr\" (UniqueName: \"kubernetes.io/projected/7689ac62-6665-4a39-80f2-1a3785898881-kube-api-access-f7jqr\") pod \"7689ac62-6665-4a39-80f2-1a3785898881\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " Nov 02 17:43:24.461750 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:24.461735 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-workload-certs\") pod \"7689ac62-6665-4a39-80f2-1a3785898881\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " Nov 02 17:43:24.461920 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:24.461806 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7689ac62-6665-4a39-80f2-1a3785898881-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7689ac62-6665-4a39-80f2-1a3785898881-istiod-ca-cert\") pod \"7689ac62-6665-4a39-80f2-1a3785898881\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " Nov 02 17:43:24.461920 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:24.461858 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-istio-envoy\") pod \"7689ac62-6665-4a39-80f2-1a3785898881\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " Nov 02 17:43:24.461920 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:24.461899 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7689ac62-6665-4a39-80f2-1a3785898881-istio-token\" (UniqueName: \"kubernetes.io/projected/7689ac62-6665-4a39-80f2-1a3785898881-istio-token\") pod \"7689ac62-6665-4a39-80f2-1a3785898881\" (UID: \"7689ac62-6665-4a39-80f2-1a3785898881\") " Nov 02 17:43:24.462556 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:24.461806 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-workload-socket" pod "7689ac62-6665-4a39-80f2-1a3785898881" (UID: "7689ac62-6665-4a39-80f2-1a3785898881"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:43:24.462691 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:24.462658 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:43:24.462691 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:24.462681 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:43:24.462823 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:24.462700 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:43:24.462823 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:24.462254 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7689ac62-6665-4a39-80f2-1a3785898881-istiod-ca-cert" pod "7689ac62-6665-4a39-80f2-1a3785898881" (UID: "7689ac62-6665-4a39-80f2-1a3785898881"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:43:24.462823 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:24.462526 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-workload-certs" pod "7689ac62-6665-4a39-80f2-1a3785898881" (UID: "7689ac62-6665-4a39-80f2-1a3785898881"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:43:24.464455 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:24.464424 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7689ac62-6665-4a39-80f2-1a3785898881-kube-api-access-f7jqr" pod "7689ac62-6665-4a39-80f2-1a3785898881" (UID: "7689ac62-6665-4a39-80f2-1a3785898881"). InnerVolumeSpecName "kube-api-access-f7jqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:43:24.464851 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:24.464824 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7689ac62-6665-4a39-80f2-1a3785898881-istio-podinfo" pod "7689ac62-6665-4a39-80f2-1a3785898881" (UID: "7689ac62-6665-4a39-80f2-1a3785898881"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:43:24.465272 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:24.465239 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7689ac62-6665-4a39-80f2-1a3785898881-istio-token" pod "7689ac62-6665-4a39-80f2-1a3785898881" (UID: "7689ac62-6665-4a39-80f2-1a3785898881"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:43:24.465420 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:24.465265 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-istio-envoy" pod "7689ac62-6665-4a39-80f2-1a3785898881" (UID: "7689ac62-6665-4a39-80f2-1a3785898881"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:43:24.564325 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:24.564277 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7jqr\" (UniqueName: \"kubernetes.io/projected/7689ac62-6665-4a39-80f2-1a3785898881-kube-api-access-f7jqr\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:43:24.564325 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:24.564328 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:43:24.564604 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:24.564345 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7689ac62-6665-4a39-80f2-1a3785898881-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:43:24.564604 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:24.564354 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7689ac62-6665-4a39-80f2-1a3785898881-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:43:24.564604 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:24.564369 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7689ac62-6665-4a39-80f2-1a3785898881-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:43:24.564604 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:24.564378 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7689ac62-6665-4a39-80f2-1a3785898881-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:43:24.884755 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:24.884721 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 17:43:24.885492 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:24.885466 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 17:43:24.886251 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:24.886226 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 17:43:25.187062 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:25.186977 4471 generic.go:356] "Generic (PLEG): container finished" podID="7689ac62-6665-4a39-80f2-1a3785898881" containerID="c556e00298ef95e9632c853cda6e55e2bf0bc1f340f88982b09be25ef1441379" exitCode=0 Nov 02 17:43:25.188787 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:25.187457 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" Nov 02 17:43:25.190577 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:25.187480 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" event={"ID":"7689ac62-6665-4a39-80f2-1a3785898881","Type":"ContainerDied","Data":"c556e00298ef95e9632c853cda6e55e2bf0bc1f340f88982b09be25ef1441379"} Nov 02 17:43:25.190684 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:25.190617 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5" event={"ID":"7689ac62-6665-4a39-80f2-1a3785898881","Type":"ContainerDied","Data":"6257aa7cab76b8858088eca1413f0d2f3e039c0a3a8897c9c3058b3bc5610e60"} Nov 02 17:43:25.190684 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:25.190653 4471 scope.go:122] "RemoveContainer" containerID="c556e00298ef95e9632c853cda6e55e2bf0bc1f340f88982b09be25ef1441379" Nov 02 17:43:25.211955 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:25.211820 4471 scope.go:122] "RemoveContainer" containerID="c556e00298ef95e9632c853cda6e55e2bf0bc1f340f88982b09be25ef1441379" Nov 02 17:43:25.212188 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:43:25.212167 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c556e00298ef95e9632c853cda6e55e2bf0bc1f340f88982b09be25ef1441379\": container with ID starting with c556e00298ef95e9632c853cda6e55e2bf0bc1f340f88982b09be25ef1441379 not found: ID does not exist" containerID="c556e00298ef95e9632c853cda6e55e2bf0bc1f340f88982b09be25ef1441379" Nov 02 17:43:25.212250 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:25.212201 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c556e00298ef95e9632c853cda6e55e2bf0bc1f340f88982b09be25ef1441379"} err="failed to get container status \"c556e00298ef95e9632c853cda6e55e2bf0bc1f340f88982b09be25ef1441379\": rpc error: code = NotFound desc = could not find container \"c556e00298ef95e9632c853cda6e55e2bf0bc1f340f88982b09be25ef1441379\": container with ID starting with c556e00298ef95e9632c853cda6e55e2bf0bc1f340f88982b09be25ef1441379 not found: ID does not exist" Nov 02 17:43:25.293639 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:25.293611 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5"] Nov 02 17:43:25.303155 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:25.303128 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8mnq5"] Nov 02 17:43:26.440819 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:26.440786 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7689ac62-6665-4a39-80f2-1a3785898881" path="/var/lib/kubelet/pods/7689ac62-6665-4a39-80f2-1a3785898881/volumes" Nov 02 17:43:31.217858 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.217820 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg"] Nov 02 17:43:31.218626 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.218606 4471 cpu_manager.go:414] "containerMap: removing container" podUID="7689ac62-6665-4a39-80f2-1a3785898881" containerName="istio-proxy" Nov 02 17:43:31.218736 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.218629 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7689ac62-6665-4a39-80f2-1a3785898881" containerName="istio-proxy" Nov 02 17:43:31.218809 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.218794 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="7689ac62-6665-4a39-80f2-1a3785898881" containerName="istio-proxy" Nov 02 17:43:31.222173 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.222150 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:43:31.377275 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.377242 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istio-envoy\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:43:31.377461 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.377400 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-workload-certs\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:43:31.377505 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.377457 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-credential-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:43:31.377505 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.377485 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istio-podinfo\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:43:31.377594 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.377516 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istio-token\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:43:31.377660 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.377639 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istiod-ca-cert\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:43:31.377698 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.377674 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-workload-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:43:31.377698 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.377691 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istio-data\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:43:31.377790 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.377717 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z82px\" (UniqueName: \"kubernetes.io/projected/60f62cc3-c3a5-4d48-995f-4844ae54e13a-kube-api-access-z82px\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:43:31.453541 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.453513 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg"] Nov 02 17:43:31.479327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.479225 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z82px\" (UniqueName: \"kubernetes.io/projected/60f62cc3-c3a5-4d48-995f-4844ae54e13a-kube-api-access-z82px\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:43:31.479327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.479281 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istio-envoy\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:43:31.479571 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.479437 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-workload-certs\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:43:31.479571 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.479468 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-credential-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:43:31.479571 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.479504 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istio-podinfo\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:43:31.479571 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.479551 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istio-token\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:43:31.479795 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.479710 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istiod-ca-cert\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:43:31.479878 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.479851 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-credential-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:43:31.480142 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.480109 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-workload-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:43:31.480240 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.480174 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-workload-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:43:31.480319 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.480237 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istio-data\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:43:31.480319 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.480280 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-workload-certs\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:43:31.480474 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.480446 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istiod-ca-cert\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:43:31.480604 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.480520 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istio-data\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:43:31.482154 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.482134 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istio-envoy\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:43:31.482760 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.482740 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istio-podinfo\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:43:31.513385 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.513354 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z82px\" (UniqueName: \"kubernetes.io/projected/60f62cc3-c3a5-4d48-995f-4844ae54e13a-kube-api-access-z82px\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:43:31.513611 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.513591 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istio-token\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:43:31.541654 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.541627 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:43:31.869800 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:43:31.869771 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f62cc3_c3a5_4d48_995f_4844ae54e13a.slice/crio-41bfe1babb911a4343d21ddb16d9ac56a67f3b1e8f166b85711254ee0c92b858 WatchSource:0}: Error finding container 41bfe1babb911a4343d21ddb16d9ac56a67f3b1e8f166b85711254ee0c92b858: Status 404 returned error can't find the container with id 41bfe1babb911a4343d21ddb16d9ac56a67f3b1e8f166b85711254ee0c92b858 Nov 02 17:43:31.872163 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.872128 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:43:31.872248 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.872201 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:43:31.872248 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.872230 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:43:31.872812 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:31.872783 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg"] Nov 02 17:43:32.238569 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:32.238483 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" event={"ID":"60f62cc3-c3a5-4d48-995f-4844ae54e13a","Type":"ContainerStarted","Data":"65e88f15f027979be09a4e14b8eec60e3f4c85cb6d234be3f5cd4e5ed37a6d7e"} Nov 02 17:43:32.238569 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:32.238552 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" event={"ID":"60f62cc3-c3a5-4d48-995f-4844ae54e13a","Type":"ContainerStarted","Data":"41bfe1babb911a4343d21ddb16d9ac56a67f3b1e8f166b85711254ee0c92b858"} Nov 02 17:43:32.304292 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:32.304220 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" podStartSLOduration=1.304202335 podStartE2EDuration="1.304202335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:32.299568486 +0000 UTC m=+829.994923959" watchObservedRunningTime="2026-11-02 17:43:32.304202335 +0000 UTC m=+829.999557810" Nov 02 17:43:32.545567 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:32.545528 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:43:32.545938 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:32.545913 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 17:43:32.546105 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:32.546077 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" podUID="60f62cc3-c3a5-4d48-995f-4844ae54e13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 17:43:33.545421 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:33.545376 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 17:43:33.545882 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:33.545537 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" podUID="60f62cc3-c3a5-4d48-995f-4844ae54e13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 17:43:34.542282 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:34.542238 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 17:43:34.542490 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:34.542341 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" podUID="60f62cc3-c3a5-4d48-995f-4844ae54e13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 17:43:34.894806 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:34.894719 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:43:35.542557 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:35.542465 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 17:43:35.542557 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:35.542524 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" podUID="60f62cc3-c3a5-4d48-995f-4844ae54e13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 17:43:36.556760 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:36.556719 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 17:43:36.557370 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:36.557331 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" podUID="60f62cc3-c3a5-4d48-995f-4844ae54e13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 17:43:37.363475 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:37.363423 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:37.374460 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:37.374418 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 02 17:43:37.488331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:37.487801 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:37.551333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:37.550864 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 17:43:37.551333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:37.550935 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" podUID="60f62cc3-c3a5-4d48-995f-4844ae54e13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 17:43:37.603444 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:37.603394 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:37.604777 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:37.604742 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:37.627569 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:37.627456 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:37.635430 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:37.635383 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:37.692332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:37.691388 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:38.102334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:38.101351 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:38.423761 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:38.423628 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:38.507343 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:38.471240 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:38.507781 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:38.479716 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:38.657609 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:38.656421 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 02 17:43:38.669336 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:38.665217 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:43:38.708514 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:38.708408 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:38.734329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:38.709209 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:38.789715 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:38.733795 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:38.790194 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:38.733824 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:38.795832 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:38.742937 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 17:43:38.835327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:38.796015 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" podUID="60f62cc3-c3a5-4d48-995f-4844ae54e13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 17:43:38.889385 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:38.888139 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 02 17:43:38.979179 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:38.979090 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:43:38.996466 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:38.996429 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:43:38.996720 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:38.996705 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:43:39.000437 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:39.000408 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:43:39.059516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:39.059483 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 17:43:39.059781 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:39.059762 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:43:39.082339 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:39.061399 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:43:39.082339 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:39.061466 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" podUID="636cf7f7-ba1a-4f54-9db0-6d8ad2909424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:39.655346 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:39.654405 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:39.684332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:39.663839 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:39.760381 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:39.737436 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 17:43:39.760381 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:39.737500 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" podUID="60f62cc3-c3a5-4d48-995f-4844ae54e13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 17:43:39.760381 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:39.753821 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:43:39.760381 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:39.754202 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 17:43:39.816254 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:39.816196 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:39.830363 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:39.816534 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:39.905741 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:39.892733 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:39.905741 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:39.893592 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:43:40.080317 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:40.080245 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:43:40.085378 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:40.080596 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" podUID="db524df0-282c-43eb-936b-53ed7d9388cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:40.344331 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:43:40.344168 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:43:40.638066 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:40.637868 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 17:43:40.638464 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:40.638430 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" podUID="60f62cc3-c3a5-4d48-995f-4844ae54e13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 17:43:40.660526 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:40.660466 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:40.714336 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:40.669755 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 02 17:43:40.715948 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:40.714998 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:43:40.715948 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:40.715080 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" podUID="ec23f89c-765f-4c30-a3e5-761ca79f0288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:40.835450 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:40.835387 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:40.872332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:40.848047 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 17:43:40.911389 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:40.848084 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:43:40.968625 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:40.848154 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:40.973341 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:40.903398 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 02 17:43:40.973341 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:40.940433 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-9m286 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:43:40.973341 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:40.969805 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" podUID="65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:40.973341 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:40.941043 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:43:40.973341 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:40.969877 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" podUID="fae7257d-c6ad-4e86-8133-aacf8110adb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:41.349501 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:41.349464 4471 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:43:41.349825 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:41.349775 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" podUID="81708d59-e82f-4259-9bf1-cec029c05c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:41.610142 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:41.607382 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 17:43:41.610422 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:41.610388 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" podUID="60f62cc3-c3a5-4d48-995f-4844ae54e13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 17:43:41.668452 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:41.668403 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:41.685758 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:41.685710 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:41.901924 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:41.880420 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 02 17:43:41.903800 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:41.903755 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:42.018710 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:42.017518 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:42.018710 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:42.017954 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": context deadline exceeded" Nov 02 17:43:42.018710 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:42.018296 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 02 17:43:42.581331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:42.577707 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 17:43:42.587481 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:42.587431 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" podUID="60f62cc3-c3a5-4d48-995f-4844ae54e13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 17:43:42.671852 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:42.671510 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 02 17:43:42.691334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:42.690112 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 02 17:43:42.748329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:42.741123 4471 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:43:42.748329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:42.741189 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" podUID="14703ce4-5dc1-43d0-8368-c217f0bab9eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:42.882025 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:42.881983 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:42.909329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:42.905373 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:43.027535 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:43.027489 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": context deadline exceeded" Nov 02 17:43:43.033339 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:43.027786 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:43:43.083440 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:43.083398 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:43.398865 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:43.398836 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:43:43.516019 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:43.515902 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:43:43.548777 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:43.548737 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 17:43:43.549413 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:43.549208 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" podUID="60f62cc3-c3a5-4d48-995f-4844ae54e13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 17:43:43.570375 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:43.570347 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:43:43.586009 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:43.585979 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:43:43.596625 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:43.596594 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:43:43.602871 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:43.602848 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 17:43:43.614330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:43.607801 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:43:45.009657 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:45.009625 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:43:45.010134 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:45.010036 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:43:45.011065 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:43:45.011043 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:44:01.365888 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.365807 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl"] Nov 02 17:44:01.369687 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.369667 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:44:01.404532 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.404498 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w"] Nov 02 17:44:01.409531 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.409504 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl"] Nov 02 17:44:01.409744 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.409654 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:44:01.440676 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.440649 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp"] Nov 02 17:44:01.445035 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.445009 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:44:01.456250 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.456228 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w"] Nov 02 17:44:01.477209 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.477174 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp"] Nov 02 17:44:01.537740 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.537600 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-credential-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-67ftl\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:44:01.537740 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.537656 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istio-token\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:44:01.537740 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.537692 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfrm4\" (UniqueName: \"kubernetes.io/projected/59aa40f6-3083-4511-bd81-0d00135a0c7b-kube-api-access-rfrm4\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:44:01.537740 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.537739 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-workload-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:44:01.538105 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.537773 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59aa40f6-3083-4511-bd81-0d00135a0c7b-istio-podinfo\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:44:01.538105 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.537870 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istio-podinfo\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:44:01.538105 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.537937 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istio-podinfo\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-67ftl\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:44:01.538105 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.537972 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istio-data\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-67ftl\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:44:01.538105 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.538002 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59aa40f6-3083-4511-bd81-0d00135a0c7b-istiod-ca-cert\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:44:01.538105 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.538055 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-credential-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:44:01.538105 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.538088 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl6v2\" (UniqueName: \"kubernetes.io/projected/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-kube-api-access-dl6v2\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-67ftl\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:44:01.538520 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.538142 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-workload-certs\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:44:01.538520 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.538205 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sm9n\" (UniqueName: \"kubernetes.io/projected/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-kube-api-access-9sm9n\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:44:01.538520 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.538232 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istiod-ca-cert\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-67ftl\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:44:01.538520 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.538325 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-workload-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-67ftl\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:44:01.538689 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.538670 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-istio-data\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:44:01.538773 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.538755 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istio-envoy\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-67ftl\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:44:01.538859 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.538833 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istio-token\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-67ftl\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:44:01.538898 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.538887 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-workload-certs\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-67ftl\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:44:01.538931 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.538917 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-workload-certs\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:44:01.539008 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.538994 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-workload-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:44:01.539044 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.539022 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59aa40f6-3083-4511-bd81-0d00135a0c7b-istio-token\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:44:01.539144 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.539123 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-istio-envoy\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:44:01.539187 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.539162 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-credential-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:44:01.539224 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.539203 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istiod-ca-cert\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:44:01.539276 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.539223 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istio-envoy\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:44:01.539394 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.539380 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istio-data\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:44:01.640653 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.640566 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istio-podinfo\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:44:01.640653 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.640614 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istio-podinfo\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-67ftl\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:44:01.640653 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.640636 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istio-data\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-67ftl\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:44:01.640653 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.640655 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59aa40f6-3083-4511-bd81-0d00135a0c7b-istiod-ca-cert\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:44:01.641014 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.640686 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-credential-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:44:01.641014 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.640795 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl6v2\" (UniqueName: \"kubernetes.io/projected/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-kube-api-access-dl6v2\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-67ftl\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:44:01.641014 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.640859 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-workload-certs\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:44:01.641014 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.640914 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sm9n\" (UniqueName: \"kubernetes.io/projected/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-kube-api-access-9sm9n\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:44:01.641014 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.640937 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istiod-ca-cert\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-67ftl\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:44:01.641014 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.640962 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-workload-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-67ftl\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:44:01.641014 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.641007 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-istio-data\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:44:01.641393 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.641029 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istio-envoy\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-67ftl\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:44:01.641393 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.641051 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istio-token\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-67ftl\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:44:01.641393 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.641088 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-workload-certs\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-67ftl\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:44:01.641393 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.641118 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-workload-certs\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:44:01.641393 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.641172 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istio-data\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-67ftl\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:44:01.641393 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.641240 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-workload-certs\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:44:01.641644 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.641421 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-istio-data\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:44:01.641644 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.641497 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59aa40f6-3083-4511-bd81-0d00135a0c7b-istiod-ca-cert\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:44:01.641644 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.641518 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-workload-certs\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-67ftl\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:44:01.641644 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.641179 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-workload-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:44:01.641644 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.641573 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59aa40f6-3083-4511-bd81-0d00135a0c7b-istio-token\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:44:01.641644 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.641640 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-istio-envoy\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:44:01.641960 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.641681 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-credential-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:44:01.641960 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.641715 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istiod-ca-cert\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:44:01.641960 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.641745 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istio-envoy\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:44:01.641960 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.641776 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istio-data\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:44:01.641960 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.641839 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-credential-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-67ftl\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:44:01.641960 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.641866 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-credential-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:44:01.641960 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.641872 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istio-token\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:44:01.641960 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.641935 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfrm4\" (UniqueName: \"kubernetes.io/projected/59aa40f6-3083-4511-bd81-0d00135a0c7b-kube-api-access-rfrm4\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:44:01.642423 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.641994 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-workload-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:44:01.642423 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.642032 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59aa40f6-3083-4511-bd81-0d00135a0c7b-istio-podinfo\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:44:01.642423 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.642251 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istiod-ca-cert\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-67ftl\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:44:01.642423 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.642362 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-credential-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:44:01.642640 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.642524 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-workload-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-67ftl\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:44:01.642811 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.642787 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-workload-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:44:01.643349 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.643067 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-workload-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:44:01.643464 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.643095 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istiod-ca-cert\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:44:01.643464 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.643236 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-credential-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-67ftl\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:44:01.643594 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.643488 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-workload-certs\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:44:01.643883 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.643842 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istio-envoy\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-67ftl\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:44:01.643883 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.643863 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istio-podinfo\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:44:01.644039 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.643930 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istio-podinfo\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-67ftl\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:44:01.644101 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.644048 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istio-data\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:44:01.644283 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.644266 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-istio-envoy\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:44:01.645079 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.645056 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59aa40f6-3083-4511-bd81-0d00135a0c7b-istio-podinfo\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:44:01.645462 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.645442 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istio-envoy\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:44:01.672630 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.672602 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istio-token\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:44:01.672934 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.672907 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istio-token\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-67ftl\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:44:01.673617 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.673599 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl6v2\" (UniqueName: \"kubernetes.io/projected/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-kube-api-access-dl6v2\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-67ftl\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:44:01.673818 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.673801 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sm9n\" (UniqueName: \"kubernetes.io/projected/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-kube-api-access-9sm9n\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:44:01.677614 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.677586 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfrm4\" (UniqueName: \"kubernetes.io/projected/59aa40f6-3083-4511-bd81-0d00135a0c7b-kube-api-access-rfrm4\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:44:01.677771 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.677754 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59aa40f6-3083-4511-bd81-0d00135a0c7b-istio-token\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:44:01.687463 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.687442 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:44:01.742200 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.742165 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:44:01.767844 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:01.767622 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:44:02.014640 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:44:02.014603 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74fa14f5_bd49_49aa_88ef_fd812f99e7bc.slice/crio-7f3d4b5d80c297cdb7b3ce941fa91a2b1aaec90a881a1fffb8d754a5a638160f WatchSource:0}: Error finding container 7f3d4b5d80c297cdb7b3ce941fa91a2b1aaec90a881a1fffb8d754a5a638160f: Status 404 returned error can't find the container with id 7f3d4b5d80c297cdb7b3ce941fa91a2b1aaec90a881a1fffb8d754a5a638160f Nov 02 17:44:02.016540 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:02.016519 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl"] Nov 02 17:44:02.017289 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:02.017258 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:44:02.017350 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:02.017340 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:44:02.017392 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:02.017374 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:44:02.096739 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:44:02.095479 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59aa40f6_3083_4511_bd81_0d00135a0c7b.slice/crio-49573f62f0f8662b46ca8b7510c3f48d0b80c60a0ad6d9adc58ac365b2ad9bad WatchSource:0}: Error finding container 49573f62f0f8662b46ca8b7510c3f48d0b80c60a0ad6d9adc58ac365b2ad9bad: Status 404 returned error can't find the container with id 49573f62f0f8662b46ca8b7510c3f48d0b80c60a0ad6d9adc58ac365b2ad9bad Nov 02 17:44:02.097984 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:02.097961 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w"] Nov 02 17:44:02.098808 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:02.098773 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:44:02.098924 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:02.098840 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:44:02.098924 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:02.098869 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:44:02.105810 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:02.105787 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp"] Nov 02 17:44:02.108465 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:44:02.108438 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bec43b9_5e3f_4c4b_b435_7926df2d53cc.slice/crio-22961d196d4d2a55622310ec757846faf4bd7b3483128f42bf6cc2a890ff44f5 WatchSource:0}: Error finding container 22961d196d4d2a55622310ec757846faf4bd7b3483128f42bf6cc2a890ff44f5: Status 404 returned error can't find the container with id 22961d196d4d2a55622310ec757846faf4bd7b3483128f42bf6cc2a890ff44f5 Nov 02 17:44:02.111405 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:02.111378 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:44:02.111516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:02.111448 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:44:02.111516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:02.111497 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:44:02.756518 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:02.756464 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" event={"ID":"7bec43b9-5e3f-4c4b-b435-7926df2d53cc","Type":"ContainerStarted","Data":"e692d256494448ad494843d34287ef5a4da85965a85eb81ff10633112e8e5142"} Nov 02 17:44:02.757030 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:02.756546 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" event={"ID":"7bec43b9-5e3f-4c4b-b435-7926df2d53cc","Type":"ContainerStarted","Data":"22961d196d4d2a55622310ec757846faf4bd7b3483128f42bf6cc2a890ff44f5"} Nov 02 17:44:02.780388 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:02.780052 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:44:02.796369 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:02.794660 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" event={"ID":"59aa40f6-3083-4511-bd81-0d00135a0c7b","Type":"ContainerStarted","Data":"2246301749e89d34745b596440102ac59124843f742d7847cd002e69573bf400"} Nov 02 17:44:02.796369 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:02.794756 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" event={"ID":"59aa40f6-3083-4511-bd81-0d00135a0c7b","Type":"ContainerStarted","Data":"49573f62f0f8662b46ca8b7510c3f48d0b80c60a0ad6d9adc58ac365b2ad9bad"} Nov 02 17:44:02.812602 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:02.812555 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" event={"ID":"74fa14f5-bd49-49aa-88ef-fd812f99e7bc","Type":"ContainerStarted","Data":"5f421836c954b905f591ab4fa3eb3f6f382bc4f91242e59962f16f1fe931cdc9"} Nov 02 17:44:02.812791 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:02.812629 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" event={"ID":"74fa14f5-bd49-49aa-88ef-fd812f99e7bc","Type":"ContainerStarted","Data":"7f3d4b5d80c297cdb7b3ce941fa91a2b1aaec90a881a1fffb8d754a5a638160f"} Nov 02 17:44:02.888346 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:02.888265 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" podStartSLOduration=1.888245484 podStartE2EDuration="1.888245484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:02.831210918 +0000 UTC m=+860.526566422" watchObservedRunningTime="2026-11-02 17:44:02.888245484 +0000 UTC m=+860.583600958" Nov 02 17:44:02.942815 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:02.942758 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" podStartSLOduration=1.942742481 podStartE2EDuration="1.942742481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:02.942503228 +0000 UTC m=+860.637858723" watchObservedRunningTime="2026-11-02 17:44:02.942742481 +0000 UTC m=+860.638097953" Nov 02 17:44:02.943706 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:02.943667 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" podStartSLOduration=1.9436572220000001 podStartE2EDuration="1.943657222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:02.889860508 +0000 UTC m=+860.585216002" watchObservedRunningTime="2026-11-02 17:44:02.943657222 +0000 UTC m=+860.639012694" Nov 02 17:44:03.693330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:03.688201 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:44:03.717362 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:03.711958 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 17:44:03.717362 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:03.712022 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" podUID="74fa14f5-bd49-49aa-88ef-fd812f99e7bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 17:44:03.742593 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:03.742560 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:44:03.744681 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:03.744635 4471 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 17:44:03.744811 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:03.744736 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" podUID="59aa40f6-3083-4511-bd81-0d00135a0c7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 17:44:03.811328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:03.810648 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 17:44:03.811328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:03.810732 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" podUID="7bec43b9-5e3f-4c4b-b435-7926df2d53cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 17:44:04.698328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:04.697683 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 17:44:04.698328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:04.697748 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" podUID="74fa14f5-bd49-49aa-88ef-fd812f99e7bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 17:44:04.742669 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:04.742629 4471 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 17:44:04.742902 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:04.742881 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" podUID="59aa40f6-3083-4511-bd81-0d00135a0c7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 17:44:04.803330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:04.793398 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 17:44:04.803330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:04.793456 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" podUID="7bec43b9-5e3f-4c4b-b435-7926df2d53cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 17:44:05.748331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:05.743444 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 17:44:05.748331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:05.743513 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" podUID="74fa14f5-bd49-49aa-88ef-fd812f99e7bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 17:44:05.773345 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:05.769145 4471 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 17:44:05.773345 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:05.769205 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" podUID="59aa40f6-3083-4511-bd81-0d00135a0c7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 17:44:05.851622 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:05.850891 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 17:44:05.851622 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:05.850943 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" podUID="7bec43b9-5e3f-4c4b-b435-7926df2d53cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 17:44:06.827333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:06.819541 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 17:44:06.827333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:06.819634 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" podUID="7bec43b9-5e3f-4c4b-b435-7926df2d53cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 17:44:06.830844 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:06.830800 4471 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 17:44:06.831066 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:06.831038 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" podUID="59aa40f6-3083-4511-bd81-0d00135a0c7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 17:44:06.872646 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:06.872595 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 17:44:06.872941 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:06.872891 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" podUID="74fa14f5-bd49-49aa-88ef-fd812f99e7bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 17:44:07.691688 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:07.688619 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 17:44:07.691688 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:07.688698 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" podUID="74fa14f5-bd49-49aa-88ef-fd812f99e7bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 17:44:07.753562 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:07.753055 4471 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 17:44:07.753562 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:07.753123 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" podUID="59aa40f6-3083-4511-bd81-0d00135a0c7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 17:44:07.779721 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:07.779682 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 17:44:07.780033 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:07.779985 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" podUID="7bec43b9-5e3f-4c4b-b435-7926df2d53cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 17:44:08.693773 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:08.693680 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 17:44:08.694368 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:08.694332 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" podUID="74fa14f5-bd49-49aa-88ef-fd812f99e7bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 17:44:08.743678 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:08.743635 4471 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 17:44:08.743834 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:08.743712 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" podUID="59aa40f6-3083-4511-bd81-0d00135a0c7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 17:44:08.770505 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:08.770457 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 17:44:08.770715 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:08.770534 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" podUID="7bec43b9-5e3f-4c4b-b435-7926df2d53cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 17:44:09.301358 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:44:09.301291 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:44:09.688821 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:09.688729 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 17:44:09.688821 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:09.688799 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" podUID="74fa14f5-bd49-49aa-88ef-fd812f99e7bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 17:44:09.769059 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:09.769018 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 17:44:09.769628 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:09.769088 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" podUID="7bec43b9-5e3f-4c4b-b435-7926df2d53cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 17:44:10.213141 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:10.213109 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:44:10.214927 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:10.214901 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:44:10.225716 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:10.225102 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:44:10.701007 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:10.700978 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:44:10.701344 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:10.701323 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:44:10.701969 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:10.701949 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:44:10.772889 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:10.772863 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:44:10.773250 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:10.773140 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:44:10.773768 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:10.773753 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:44:13.696332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:13.695423 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 02 17:44:13.696865 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:13.696672 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:13.781330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:13.780765 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:44:13.822331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:13.822197 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:13.846332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:13.843986 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:13.895091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:13.895045 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 17:44:13.978493 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:13.978391 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:14.749340 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:14.749268 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:14.753077 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:14.753037 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:14.853131 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:14.807536 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:14.853131 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:14.846471 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:14.853131 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:14.846754 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 02 17:44:14.898754 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:14.898709 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:15.006390 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:14.976449 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded" Nov 02 17:44:15.006390 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:14.980438 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:15.059329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:15.033957 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:44:15.061656 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:15.061626 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:44:15.061903 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:15.061886 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:44:15.062077 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:15.062064 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 17:44:15.062213 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:15.062200 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:44:15.137330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:15.137227 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:44:15.137330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:15.137291 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:44:15.881329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:15.842796 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 02 17:44:15.982329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:15.903021 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:15.982542 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:15.904133 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:15.982620 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:15.904164 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:44:16.021330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:15.904184 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": context deadline exceeded" Nov 02 17:44:16.070570 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:15.904203 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 02 17:44:16.075226 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:15.920425 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:44:16.075491 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:16.075458 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" podUID="0316e437-096a-4330-b71c-3b33b139c4d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:16.075619 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:16.057685 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:16.100206 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:16.100170 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:44:16.100547 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:16.100515 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" podUID="60f62cc3-c3a5-4d48-995f-4844ae54e13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:16.175891 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:16.175792 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:44:16.176171 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:16.176135 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" podUID="7d2e5cde-97d4-4e6a-a564-55693ade99ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:16.834874 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:16.834833 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v"] Nov 02 17:44:17.030683 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.029547 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 02 17:44:17.043414 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.030077 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 17:44:17.076345 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.044116 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 02 17:44:17.076345 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.045185 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:17.076345 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.046370 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:44:17.126363 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.076761 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": context deadline exceeded" Nov 02 17:44:17.126363 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.080161 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 02 17:44:17.126363 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.097559 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 02 17:44:17.309975 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.309943 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istiod-ca-cert\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:44:17.310530 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.310500 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-credential-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:44:17.321640 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.321608 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-workload-certs\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:44:17.322011 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.321986 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istio-podinfo\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:44:17.322175 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.322153 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istio-envoy\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:44:17.322339 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.322323 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istio-token\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:44:17.322503 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.322488 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmvqj\" (UniqueName: \"kubernetes.io/projected/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-kube-api-access-pmvqj\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:44:17.322719 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.322697 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-workload-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:44:17.322872 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.322857 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istio-data\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:44:17.358350 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.358146 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v"] Nov 02 17:44:17.386872 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.386774 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:17.460021 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.433013 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-workload-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:44:17.460021 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.445400 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istio-data\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:44:17.460021 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.445454 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istiod-ca-cert\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:44:17.460021 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.445578 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-credential-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:44:17.460021 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.445649 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-workload-certs\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:44:17.460021 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.445793 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istio-podinfo\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:44:17.460021 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.445827 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istio-envoy\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:44:17.460021 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.445872 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istio-token\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:44:17.460021 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.445936 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmvqj\" (UniqueName: \"kubernetes.io/projected/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-kube-api-access-pmvqj\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:44:17.460021 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.433552 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-workload-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:44:17.460021 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.458947 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istio-data\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:44:17.460021 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.459771 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istiod-ca-cert\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:44:17.462399 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.462371 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-credential-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:44:17.470760 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.470716 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-workload-certs\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:44:17.528743 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.521614 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmvqj\" (UniqueName: \"kubernetes.io/projected/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-kube-api-access-pmvqj\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:44:17.529983 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.529956 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:44:17.535607 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.533226 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istio-podinfo\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:44:17.538283 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.538261 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istio-envoy\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:44:17.542933 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.542876 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istio-token\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:44:17.573234 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.573184 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:17.585749 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.585708 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:44:17.646002 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:17.645910 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 17:44:18.036167 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:18.036137 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:18.057227 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:18.055525 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 02 17:44:18.057227 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:18.055860 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 17:44:18.129331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:18.128690 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:18.161506 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:18.161456 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:18.342376 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:18.341910 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:44:18.381740 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:18.381701 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:44:18.416386 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:18.400575 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:44:18.416386 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:18.401028 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:44:18.416386 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:18.402503 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:44:18.634710 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:18.632233 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:44:18.634710 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:18.632339 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:44:18.634710 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:18.632392 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:44:18.640681 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:18.640657 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v"] Nov 02 17:44:19.522943 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:19.522905 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" event={"ID":"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd","Type":"ContainerStarted","Data":"62b0cdf9b85ee92dd073ded7ffbc21c0851e1510c9098b500896204ed1179329"} Nov 02 17:44:19.523341 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:19.522971 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" event={"ID":"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd","Type":"ContainerStarted","Data":"fe4f019f00d4a214930bf87f3ea22c26132320b8c774c58fa97b2b9f06526492"} Nov 02 17:44:19.590473 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:19.590437 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:44:19.592209 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:19.591941 4471 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 17:44:19.592209 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:19.591994 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" podUID="04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 17:44:19.617643 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:19.617582 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" podStartSLOduration=3.6175620569999998 podStartE2EDuration="3.617562057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:19.616371831 +0000 UTC m=+877.311727321" watchObservedRunningTime="2026-11-02 17:44:19.617562057 +0000 UTC m=+877.312917531" Nov 02 17:44:20.590256 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:20.590216 4471 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 17:44:20.590690 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:20.590277 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" podUID="04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 17:44:21.589855 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:21.589815 4471 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 17:44:21.590034 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:21.589877 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" podUID="04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 17:44:22.589818 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:22.589783 4471 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 17:44:22.590210 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:22.589839 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" podUID="04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 17:44:23.590196 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:23.590166 4471 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 17:44:23.590648 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:23.590218 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" podUID="04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 17:44:24.590076 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:24.590042 4471 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 17:44:24.590244 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:24.590107 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" podUID="04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 17:44:25.590584 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:25.590544 4471 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 17:44:25.590990 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:25.590611 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" podUID="04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 17:44:26.594912 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:26.594878 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:44:26.595388 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:26.595334 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:44:26.596003 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:26.595987 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:44:34.470844 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:34.470811 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl"] Nov 02 17:44:34.471450 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:34.471049 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" podUID="398b95b8-a6ae-442f-ba56-d9c77e4b2050" containerName="istio-proxy" containerID="cri-o://501e013e74fce830a2279f8b4a8bb07af4b2a6a70172d11e33fd1eb5872157a5" gracePeriod=30 Nov 02 17:44:39.183318 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:44:39.183267 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:44:39.661607 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:39.661568 4471 generic.go:356] "Generic (PLEG): container finished" podID="398b95b8-a6ae-442f-ba56-d9c77e4b2050" containerID="501e013e74fce830a2279f8b4a8bb07af4b2a6a70172d11e33fd1eb5872157a5" exitCode=0 Nov 02 17:44:39.661821 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:39.661796 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" event={"ID":"398b95b8-a6ae-442f-ba56-d9c77e4b2050","Type":"ContainerDied","Data":"501e013e74fce830a2279f8b4a8bb07af4b2a6a70172d11e33fd1eb5872157a5"} Nov 02 17:44:39.735214 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:39.735146 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:44:39.876179 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:39.876144 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istio-podinfo\") pod \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " Nov 02 17:44:39.876426 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:39.876213 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/398b95b8-a6ae-442f-ba56-d9c77e4b2050-kube-api-access-vwqdq\" (UniqueName: \"kubernetes.io/projected/398b95b8-a6ae-442f-ba56-d9c77e4b2050-kube-api-access-vwqdq\") pod \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " Nov 02 17:44:39.876426 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:39.876249 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-workload-socket\") pod \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " Nov 02 17:44:39.876426 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:39.876270 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istio-data\") pod \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " Nov 02 17:44:39.876426 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:39.876288 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istiod-ca-cert\") pod \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " Nov 02 17:44:39.876426 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:39.876327 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istio-token\" (UniqueName: \"kubernetes.io/projected/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istio-token\") pod \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " Nov 02 17:44:39.876623 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:39.876573 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-credential-socket\") pod \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " Nov 02 17:44:39.876675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:39.876658 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istio-envoy\") pod \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " Nov 02 17:44:39.876764 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:39.876744 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-workload-certs\") pod \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\" (UID: \"398b95b8-a6ae-442f-ba56-d9c77e4b2050\") " Nov 02 17:44:39.877191 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:39.877156 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-workload-certs" pod "398b95b8-a6ae-442f-ba56-d9c77e4b2050" (UID: "398b95b8-a6ae-442f-ba56-d9c77e4b2050"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:44:39.877667 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:39.877640 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istiod-ca-cert" pod "398b95b8-a6ae-442f-ba56-d9c77e4b2050" (UID: "398b95b8-a6ae-442f-ba56-d9c77e4b2050"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:44:39.877863 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:39.877848 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:44:39.877952 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:39.877929 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-workload-socket" pod "398b95b8-a6ae-442f-ba56-d9c77e4b2050" (UID: "398b95b8-a6ae-442f-ba56-d9c77e4b2050"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:44:39.878254 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:39.878193 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istio-data" pod "398b95b8-a6ae-442f-ba56-d9c77e4b2050" (UID: "398b95b8-a6ae-442f-ba56-d9c77e4b2050"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:44:39.878472 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:39.878406 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-credential-socket" pod "398b95b8-a6ae-442f-ba56-d9c77e4b2050" (UID: "398b95b8-a6ae-442f-ba56-d9c77e4b2050"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:44:39.879091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:39.879058 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/398b95b8-a6ae-442f-ba56-d9c77e4b2050-kube-api-access-vwqdq" pod "398b95b8-a6ae-442f-ba56-d9c77e4b2050" (UID: "398b95b8-a6ae-442f-ba56-d9c77e4b2050"). InnerVolumeSpecName "kube-api-access-vwqdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:44:39.880253 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:39.880227 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istio-token" pod "398b95b8-a6ae-442f-ba56-d9c77e4b2050" (UID: "398b95b8-a6ae-442f-ba56-d9c77e4b2050"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:44:39.881127 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:39.881102 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istio-podinfo" pod "398b95b8-a6ae-442f-ba56-d9c77e4b2050" (UID: "398b95b8-a6ae-442f-ba56-d9c77e4b2050"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:44:39.881658 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:39.881622 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istio-envoy" pod "398b95b8-a6ae-442f-ba56-d9c77e4b2050" (UID: "398b95b8-a6ae-442f-ba56-d9c77e4b2050"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:44:39.979867 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:39.979829 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:44:39.979867 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:39.979864 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:44:39.980083 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:39.979878 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:44:39.980083 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:39.979889 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwqdq\" (UniqueName: \"kubernetes.io/projected/398b95b8-a6ae-442f-ba56-d9c77e4b2050-kube-api-access-vwqdq\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:44:39.980083 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:39.979901 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:44:39.980083 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:39.979912 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:44:39.980083 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:39.979924 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:44:39.980083 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:39.979934 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/398b95b8-a6ae-442f-ba56-d9c77e4b2050-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:44:40.669562 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:40.669536 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" Nov 02 17:44:40.669971 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:40.669575 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl" event={"ID":"398b95b8-a6ae-442f-ba56-d9c77e4b2050","Type":"ContainerDied","Data":"ee45f63b60ef9a23581c7c646c87d9c0c412cf9ebadac6c79faf9970fd61475c"} Nov 02 17:44:40.669971 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:40.669633 4471 scope.go:122] "RemoveContainer" containerID="501e013e74fce830a2279f8b4a8bb07af4b2a6a70172d11e33fd1eb5872157a5" Nov 02 17:44:40.747884 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:40.747818 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl"] Nov 02 17:44:40.760081 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:40.760052 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5zqhl"] Nov 02 17:44:42.442674 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:42.442634 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="398b95b8-a6ae-442f-ba56-d9c77e4b2050" path="/var/lib/kubelet/pods/398b95b8-a6ae-442f-ba56-d9c77e4b2050/volumes" Nov 02 17:44:44.160632 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:44.160586 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:44:46.416406 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.416366 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk"] Nov 02 17:44:46.436624 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.436586 4471 cpu_manager.go:414] "containerMap: removing container" podUID="398b95b8-a6ae-442f-ba56-d9c77e4b2050" containerName="istio-proxy" Nov 02 17:44:46.436849 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.436831 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="398b95b8-a6ae-442f-ba56-d9c77e4b2050" containerName="istio-proxy" Nov 02 17:44:46.437159 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.437141 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="398b95b8-a6ae-442f-ba56-d9c77e4b2050" containerName="istio-proxy" Nov 02 17:44:46.516332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.512676 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:44:46.584330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.583196 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-credential-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:44:46.584330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.583299 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istio-token\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:44:46.584330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.583395 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istio-podinfo\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:44:46.584330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.583537 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istio-envoy\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:44:46.584330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.583630 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istio-data\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:44:46.584330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.583677 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-workload-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:44:46.584330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.583721 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-workload-certs\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:44:46.584330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.583762 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgdhp\" (UniqueName: \"kubernetes.io/projected/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-kube-api-access-kgdhp\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:44:46.584330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.583874 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istiod-ca-cert\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:44:46.686612 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.686576 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istio-podinfo\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:44:46.699539 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.699507 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istio-envoy\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:44:46.699848 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.699827 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istio-data\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:44:46.700052 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.700020 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-workload-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:44:46.725480 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.706089 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-workload-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:44:46.725730 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.706391 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istio-data\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:44:46.725910 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.725871 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-workload-certs\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:44:46.726081 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.726052 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgdhp\" (UniqueName: \"kubernetes.io/projected/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-kube-api-access-kgdhp\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:44:46.726525 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.726501 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istiod-ca-cert\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:44:46.726849 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.726833 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-credential-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:44:46.799735 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.727358 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-credential-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:44:46.799735 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.752549 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-workload-certs\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:44:46.799735 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.753611 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istiod-ca-cert\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:44:46.838567 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.838523 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istio-token\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:44:46.905342 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.879799 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7"] Nov 02 17:44:46.905342 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.884256 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istio-token\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:44:46.905342 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.890069 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istio-podinfo\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:44:46.905342 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.892658 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istio-envoy\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:44:46.919550 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.919513 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk"] Nov 02 17:44:46.926386 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.919795 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7"] Nov 02 17:44:46.931330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.920007 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgdhp\" (UniqueName: \"kubernetes.io/projected/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-kube-api-access-kgdhp\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:44:46.935425 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.920113 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:44:46.992660 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.991401 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:46.992660 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.991868 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 02 17:44:46.992660 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:46.992478 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 02 17:44:47.075775 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.054272 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-workload-certs\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:44:47.083227 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.072205 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:47.083227 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.072237 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 17:44:47.083227 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.072270 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:47.084686 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.084657 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istio-data\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:44:47.084948 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.084926 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-credential-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:44:47.085148 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.085128 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istio-token\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:44:47.085372 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.085351 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istio-envoy\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:44:47.085640 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.085620 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfn4x\" (UniqueName: \"kubernetes.io/projected/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-kube-api-access-hfn4x\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:44:47.085841 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.085824 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istiod-ca-cert\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:44:47.086040 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.086023 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-workload-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:44:47.093573 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.093546 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istio-podinfo\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:44:47.135788 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.134426 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:47.143599 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.143568 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:44:47.202543 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.199170 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istio-token\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:44:47.202543 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.199259 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istio-envoy\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:44:47.202543 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.199339 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfn4x\" (UniqueName: \"kubernetes.io/projected/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-kube-api-access-hfn4x\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:44:47.202543 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.199396 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istiod-ca-cert\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:44:47.202543 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.199441 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-workload-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:44:47.202543 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.199482 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istio-podinfo\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:44:47.202543 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.199540 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-workload-certs\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:44:47.202543 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.199640 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istio-data\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:44:47.202543 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.199684 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-credential-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:44:47.202543 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.200114 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-credential-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:44:47.205451 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.203727 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-workload-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:44:47.205451 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.204211 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istiod-ca-cert\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:44:47.206219 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.206195 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-workload-certs\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:44:47.220032 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.219962 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istio-data\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:44:47.220427 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.220361 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istio-podinfo\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:44:47.246666 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.237831 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istio-envoy\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:44:47.281010 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.280935 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istio-token\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:44:47.312351 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.302366 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfn4x\" (UniqueName: \"kubernetes.io/projected/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-kube-api-access-hfn4x\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:44:47.361536 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.361492 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": context deadline exceeded" Nov 02 17:44:47.631474 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.631423 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:47.700754 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:47.700715 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:44:48.004915 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:48.004874 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:48.005583 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:48.005538 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:44:48.006101 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:48.006078 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 02 17:44:48.006954 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:48.006937 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:44:48.007156 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:48.007141 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:44:48.261584 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:48.007878 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:44:48.261584 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:48.058924 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:48.261584 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:48.090100 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 02 17:44:48.261584 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:48.090133 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 17:44:48.261584 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:48.224456 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:44:48.261584 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:48.224512 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:44:48.261584 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:48.091504 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": context deadline exceeded" Nov 02 17:44:48.261584 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:48.184458 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 02 17:44:48.261584 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:48.248431 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 17:44:48.261584 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:48.248500 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:44:48.505332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:48.417473 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:48.570383 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:48.569443 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded" Nov 02 17:44:48.827700 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:48.827583 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/health\": context deadline exceeded" Nov 02 17:44:49.119409 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:49.111381 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 02 17:44:49.271336 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:49.269264 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:49.272078 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:49.272039 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:49.277332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:49.272848 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:49.298340 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:49.294922 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 02 17:44:49.316923 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:49.316880 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 17:44:49.346330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:49.318882 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf" podUID="ad9d1c12-a624-47ee-80b2-db72091ec1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:49.482832 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:49.482784 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 02 17:44:49.586333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:49.584383 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf" podUID="ad9d1c12-a624-47ee-80b2-db72091ec1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:50.139558 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:44:50.139517 4471 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.696s" Nov 02 17:44:50.203471 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:50.203423 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 02 17:44:50.321797 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:50.321738 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:50.335592 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:50.335532 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:44:50.337022 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:50.336972 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:50.535387 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:50.493911 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 02 17:44:50.535387 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:50.494190 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 17:44:50.535387 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:50.534295 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf" podUID="ad9d1c12-a624-47ee-80b2-db72091ec1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:50.547544 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:50.547101 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:44:50.547544 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:50.547198 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:44:50.547544 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:50.547239 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:44:50.587384 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:50.583937 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk"] Nov 02 17:44:50.587384 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:50.584037 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:44:50.587384 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:50.584096 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 17:44:50.587384 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:50.584131 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:44:50.587384 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:50.584162 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:44:50.587384 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:50.584205 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf" Nov 02 17:44:50.665386 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:50.665351 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v25sf" Nov 02 17:44:51.191402 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:51.189653 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:44:51.199621 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:51.199590 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:44:51.205554 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:51.205525 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:44:51.209969 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:51.209920 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:44:51.210172 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:51.210147 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:44:51.210374 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:51.210354 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:44:51.269047 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:51.268891 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7"] Nov 02 17:44:51.374255 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:51.374216 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" event={"ID":"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324","Type":"ContainerStarted","Data":"db5da7d150d9657a69000f52f423a6e997e39321395780cf57a27248785e73e9"} Nov 02 17:44:51.374585 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:51.374561 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" event={"ID":"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324","Type":"ContainerStarted","Data":"81a130821a26cf3a4b916e360ddc6ceeb810745b8a7bbd934737caeea443bcd7"} Nov 02 17:44:51.378837 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:51.378704 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" event={"ID":"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb","Type":"ContainerStarted","Data":"5ca90e8154f855226f92cc4d9b027d5fca2dadf3ba94631369180487899cf31b"} Nov 02 17:44:51.435183 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:51.435112 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" podStartSLOduration=5.435093556 podStartE2EDuration="5.435093556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:51.430924438 +0000 UTC m=+909.126279908" watchObservedRunningTime="2026-11-02 17:44:51.435093556 +0000 UTC m=+909.130449030" Nov 02 17:44:52.146726 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:52.146683 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:44:52.149110 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:52.149076 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 17:44:52.149237 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:52.149136 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" podUID="913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 17:44:52.416409 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:52.416286 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" event={"ID":"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb","Type":"ContainerStarted","Data":"aae29c16df8b2851c59533259ff74bf1c97cd4f10dac9ef546f36abf7e79172d"} Nov 02 17:44:52.517530 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:52.517089 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" podStartSLOduration=6.517074147 podStartE2EDuration="6.517074147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:52.486794048 +0000 UTC m=+910.182149553" watchObservedRunningTime="2026-11-02 17:44:52.517074147 +0000 UTC m=+910.212429620" Nov 02 17:44:52.713585 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:52.705229 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:44:52.716146 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:52.715827 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 17:44:52.716146 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:52.715889 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" podUID="9303eeae-3eca-4a5e-b2b2-c7671ecc41cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 17:44:53.144982 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:53.144942 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 17:44:53.145152 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:53.145000 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" podUID="913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 17:44:53.705697 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:53.705662 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 17:44:53.706895 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:53.706855 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" podUID="9303eeae-3eca-4a5e-b2b2-c7671ecc41cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 17:44:54.144482 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:54.144450 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 17:44:54.144646 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:54.144505 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" podUID="913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 17:44:54.706629 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:54.706589 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 17:44:54.707217 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:54.707182 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" podUID="9303eeae-3eca-4a5e-b2b2-c7671ecc41cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 17:44:55.145436 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:55.145402 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 17:44:55.145662 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:55.145640 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" podUID="913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 17:44:55.703483 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:55.703445 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 17:44:55.703764 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:55.703738 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" podUID="9303eeae-3eca-4a5e-b2b2-c7671ecc41cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 17:44:56.144737 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:56.144702 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 17:44:56.145153 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:56.144757 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" podUID="913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 17:44:56.708802 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:56.708759 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 17:44:56.709097 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:56.709067 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" podUID="9303eeae-3eca-4a5e-b2b2-c7671ecc41cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 17:44:57.144553 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:57.144519 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 17:44:57.144792 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:57.144772 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" podUID="913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 17:44:57.711331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:57.711257 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 17:44:57.711549 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:57.711347 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" podUID="9303eeae-3eca-4a5e-b2b2-c7671ecc41cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 17:44:58.145009 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:58.144971 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 17:44:58.145573 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:58.145037 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" podUID="913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 17:44:58.702662 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:58.702619 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 17:44:58.702825 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:58.702687 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" podUID="9303eeae-3eca-4a5e-b2b2-c7671ecc41cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 17:44:59.224199 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:59.224172 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:44:59.225218 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:59.225191 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:44:59.226612 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:59.226549 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:44:59.706699 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:59.706670 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:44:59.707058 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:59.707038 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:44:59.707931 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:44:59.707910 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:45:03.685646 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:03.685616 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6806-b75dce"] Nov 02 17:45:03.689796 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:03.689779 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6806-b75dce" Nov 02 17:45:03.714496 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:03.714458 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6806-b75dce"] Nov 02 17:45:03.719760 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:03.719731 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9tmjt\"" Nov 02 17:45:03.731716 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:03.731689 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfjmb\" (UniqueName: \"kubernetes.io/projected/4a7b68c6-264d-4724-ba6b-be3ffea2fa4a-kube-api-access-lfjmb\") pod \"test-curl-6806-b75dce\" (UID: \"4a7b68c6-264d-4724-ba6b-be3ffea2fa4a\") " pod="opendatahub/test-curl-6806-b75dce" Nov 02 17:45:03.872835 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:03.872790 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfjmb\" (UniqueName: \"kubernetes.io/projected/4a7b68c6-264d-4724-ba6b-be3ffea2fa4a-kube-api-access-lfjmb\") pod \"test-curl-6806-b75dce\" (UID: \"4a7b68c6-264d-4724-ba6b-be3ffea2fa4a\") " pod="opendatahub/test-curl-6806-b75dce" Nov 02 17:45:03.904368 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:03.904280 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6803-59249b"] Nov 02 17:45:03.912687 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:03.912643 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfjmb\" (UniqueName: \"kubernetes.io/projected/4a7b68c6-264d-4724-ba6b-be3ffea2fa4a-kube-api-access-lfjmb\") pod \"test-curl-6806-b75dce\" (UID: \"4a7b68c6-264d-4724-ba6b-be3ffea2fa4a\") " pod="opendatahub/test-curl-6806-b75dce" Nov 02 17:45:03.915406 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:03.915379 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6803-59249b" Nov 02 17:45:03.924118 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:03.924089 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6803-59249b"] Nov 02 17:45:03.975393 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:03.975291 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chhdp\" (UniqueName: \"kubernetes.io/projected/17d2f69e-081c-420b-bded-9454a7db97fe-kube-api-access-chhdp\") pod \"test-curl-6803-59249b\" (UID: \"17d2f69e-081c-420b-bded-9454a7db97fe\") " pod="opendatahub/test-curl-6803-59249b" Nov 02 17:45:04.001935 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:04.001909 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6806-b75dce" Nov 02 17:45:04.077012 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:04.076297 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chhdp\" (UniqueName: \"kubernetes.io/projected/17d2f69e-081c-420b-bded-9454a7db97fe-kube-api-access-chhdp\") pod \"test-curl-6803-59249b\" (UID: \"17d2f69e-081c-420b-bded-9454a7db97fe\") " pod="opendatahub/test-curl-6803-59249b" Nov 02 17:45:04.101575 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:04.101548 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chhdp\" (UniqueName: \"kubernetes.io/projected/17d2f69e-081c-420b-bded-9454a7db97fe-kube-api-access-chhdp\") pod \"test-curl-6803-59249b\" (UID: \"17d2f69e-081c-420b-bded-9454a7db97fe\") " pod="opendatahub/test-curl-6803-59249b" Nov 02 17:45:04.237031 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:04.236954 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6803-59249b" Nov 02 17:45:04.309731 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:45:04.307071 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a7b68c6_264d_4724_ba6b_be3ffea2fa4a.slice/crio-acfb3ca280645bfdcb433be212a6c0902adf53d469f1236f8c7fe29528ff220f WatchSource:0}: Error finding container acfb3ca280645bfdcb433be212a6c0902adf53d469f1236f8c7fe29528ff220f: Status 404 returned error can't find the container with id acfb3ca280645bfdcb433be212a6c0902adf53d469f1236f8c7fe29528ff220f Nov 02 17:45:04.314478 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:04.314449 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6806-b75dce"] Nov 02 17:45:04.316758 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:04.316734 4471 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:45:04.536294 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:45:04.536251 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17d2f69e_081c_420b_bded_9454a7db97fe.slice/crio-ad8141b95d065239604c6538e54c92005721dd1afa1452696506e567c7445ecf WatchSource:0}: Error finding container ad8141b95d065239604c6538e54c92005721dd1afa1452696506e567c7445ecf: Status 404 returned error can't find the container with id ad8141b95d065239604c6538e54c92005721dd1afa1452696506e567c7445ecf Nov 02 17:45:04.538979 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:04.538956 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6803-59249b"] Nov 02 17:45:04.636596 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:04.636562 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6803-59249b" event={"ID":"17d2f69e-081c-420b-bded-9454a7db97fe","Type":"ContainerStarted","Data":"ad8141b95d065239604c6538e54c92005721dd1afa1452696506e567c7445ecf"} Nov 02 17:45:04.637877 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:04.637849 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6806-b75dce" event={"ID":"4a7b68c6-264d-4724-ba6b-be3ffea2fa4a","Type":"ContainerStarted","Data":"acfb3ca280645bfdcb433be212a6c0902adf53d469f1236f8c7fe29528ff220f"} Nov 02 17:45:07.759367 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:07.748443 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6803-59249b" event={"ID":"17d2f69e-081c-420b-bded-9454a7db97fe","Type":"ContainerStarted","Data":"688ffa415b9c0456e9a78b592b2cd15ea929743bacdc850205f47a45e50f11b1"} Nov 02 17:45:07.792391 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:07.781199 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6806-b75dce" event={"ID":"4a7b68c6-264d-4724-ba6b-be3ffea2fa4a","Type":"ContainerStarted","Data":"d80a3874a94d69ab02e8a587b4198b8dac2f7c1006b9a2534bb967b17234227a"} Nov 02 17:45:07.911524 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:07.911063 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6806-b75dce" podStartSLOduration=2.508002283 podStartE2EDuration="4.911032875s" totalImagesPullingTime="2.403030592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:07.890554231 +0000 UTC m=+925.585909698" watchObservedRunningTime="2026-11-02 17:45:07.911032875 +0000 UTC m=+925.606388360" Nov 02 17:45:07.936409 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:07.916811 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6803-59249b" podStartSLOduration=2.437938121 podStartE2EDuration="4.916788293s" totalImagesPullingTime="2.478850172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:07.846590521 +0000 UTC m=+925.541946064" watchObservedRunningTime="2026-11-02 17:45:07.916788293 +0000 UTC m=+925.612143770" Nov 02 17:45:08.575397 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:08.575359 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx"] Nov 02 17:45:08.585131 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:08.585094 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:45:08.612611 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:08.608980 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx"] Nov 02 17:45:08.616052 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:08.616028 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-556cd8a1-openshift-default-dockercfg-6zgbq\"" Nov 02 17:45:08.695319 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:08.691795 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-workload-certs\") pod \"e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:45:08.695319 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:08.691937 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istio-envoy\") pod \"e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:45:08.695319 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:08.692085 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istio-podinfo\") pod \"e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:45:08.695319 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:08.692168 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istio-data\") pod \"e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:45:08.695319 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:08.692195 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-workload-socket\") pod \"e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:45:08.695319 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:08.692238 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istiod-ca-cert\") pod \"e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:45:08.695319 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:08.692291 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlpwf\" (UniqueName: \"kubernetes.io/projected/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-kube-api-access-xlpwf\") pod \"e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:45:08.695319 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:08.692377 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-credential-socket\") pod \"e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:45:08.695319 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:08.692478 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istio-token\") pod \"e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:45:08.769326 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:08.759110 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6803-59249b"] Nov 02 17:45:08.794359 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:08.783076 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6803-59249b"] Nov 02 17:45:08.798993 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:08.795355 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-credential-socket\") pod \"e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:45:08.798993 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:08.795435 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istio-token\") pod \"e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:45:08.798993 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:08.795532 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-workload-certs\") pod \"e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:45:08.798993 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:08.795596 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istio-envoy\") pod \"e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:45:08.798993 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:08.795728 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istio-podinfo\") pod \"e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:45:08.798993 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:08.795801 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istio-data\") pod \"e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:45:08.798993 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:08.795826 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-workload-socket\") pod \"e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:45:08.798993 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:08.795875 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istiod-ca-cert\") pod \"e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:45:08.798993 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:08.795998 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlpwf\" (UniqueName: \"kubernetes.io/projected/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-kube-api-access-xlpwf\") pod \"e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:45:08.798993 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:08.796944 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-credential-socket\") pod \"e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:45:08.798993 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:08.797573 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-workload-certs\") pod \"e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:45:08.799685 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:08.799527 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-workload-socket\") pod \"e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:45:08.819993 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:08.799788 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istio-data\") pod \"e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:45:08.819993 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:08.799942 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istiod-ca-cert\") pod \"e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:45:08.869949 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:08.869181 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istio-envoy\") pod \"e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:45:08.870703 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:08.870648 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istio-podinfo\") pod \"e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:45:08.874652 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:08.874570 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6806-b75dce"] Nov 02 17:45:08.879753 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:08.879717 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6806-b75dce"] Nov 02 17:45:09.005332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:09.003937 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istio-token\") pod \"e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:45:09.108718 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:09.106341 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlpwf\" (UniqueName: \"kubernetes.io/projected/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-kube-api-access-xlpwf\") pod \"e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:45:09.240497 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:09.240416 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:45:09.338913 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:45:09.323781 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:45:09.722042 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:09.718439 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx"] Nov 02 17:45:09.734103 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:09.728889 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:45:09.734103 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:09.728990 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:45:09.734103 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:09.729036 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:45:09.871395 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:09.868471 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6806-b75dce" podUID="4a7b68c6-264d-4724-ba6b-be3ffea2fa4a" containerName="test-curl-6806-b75dce" containerID="cri-o://d80a3874a94d69ab02e8a587b4198b8dac2f7c1006b9a2534bb967b17234227a" gracePeriod=2 Nov 02 17:45:09.871395 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:09.868660 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" event={"ID":"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9","Type":"ContainerStarted","Data":"5449ba17dd9f249d5a9f441c4123d5fc833f2c60c8d6c593076d7dcb3c4439a1"} Nov 02 17:45:09.890341 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:45:09.884619 4471 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6803-59249b\" is forbidden: User \"system:node:ip-10-0-142-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-26.ec2.internal' and this object" podUID="17d2f69e-081c-420b-bded-9454a7db97fe" pod="opendatahub/test-curl-6803-59249b" Nov 02 17:45:09.890341 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:09.888800 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6803-59249b" podUID="17d2f69e-081c-420b-bded-9454a7db97fe" containerName="test-curl-6803-59249b" containerID="cri-o://688ffa415b9c0456e9a78b592b2cd15ea929743bacdc850205f47a45e50f11b1" gracePeriod=2 Nov 02 17:45:09.901336 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:45:09.900508 4471 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6806-b75dce\" is forbidden: User \"system:node:ip-10-0-142-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-26.ec2.internal' and this object" podUID="4a7b68c6-264d-4724-ba6b-be3ffea2fa4a" pod="opendatahub/test-curl-6806-b75dce" Nov 02 17:45:09.922689 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:45:09.922640 4471 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6803-59249b\" is forbidden: User \"system:node:ip-10-0-142-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-26.ec2.internal' and this object" podUID="17d2f69e-081c-420b-bded-9454a7db97fe" pod="opendatahub/test-curl-6803-59249b" Nov 02 17:45:09.937027 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:45:09.936970 4471 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6806-b75dce\" is forbidden: User \"system:node:ip-10-0-142-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-26.ec2.internal' and this object" podUID="4a7b68c6-264d-4724-ba6b-be3ffea2fa4a" pod="opendatahub/test-curl-6806-b75dce" Nov 02 17:45:10.867352 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:10.867290 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" event={"ID":"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9","Type":"ContainerStarted","Data":"21a78147746a371b82dbfd9c3e52c5e48cab2a88aa7df734c4de840afc6c2022"} Nov 02 17:45:10.973620 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:10.973567 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" podStartSLOduration=2.973554502 podStartE2EDuration="2.973554502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:10.963746259 +0000 UTC m=+928.659101776" watchObservedRunningTime="2026-11-02 17:45:10.973554502 +0000 UTC m=+928.668909975" Nov 02 17:45:11.243879 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:11.241896 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:45:11.248668 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:11.248641 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:45:11.886279 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:11.886242 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:45:11.887321 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:11.887279 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:45:12.187759 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.187733 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6803-59249b" Nov 02 17:45:12.191391 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.191371 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6806-b75dce" Nov 02 17:45:12.239877 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.239846 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17d2f69e-081c-420b-bded-9454a7db97fe-kube-api-access-chhdp\" (UniqueName: \"kubernetes.io/projected/17d2f69e-081c-420b-bded-9454a7db97fe-kube-api-access-chhdp\") pod \"17d2f69e-081c-420b-bded-9454a7db97fe\" (UID: \"17d2f69e-081c-420b-bded-9454a7db97fe\") " Nov 02 17:45:12.240003 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.239948 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a7b68c6-264d-4724-ba6b-be3ffea2fa4a-kube-api-access-lfjmb\" (UniqueName: \"kubernetes.io/projected/4a7b68c6-264d-4724-ba6b-be3ffea2fa4a-kube-api-access-lfjmb\") pod \"4a7b68c6-264d-4724-ba6b-be3ffea2fa4a\" (UID: \"4a7b68c6-264d-4724-ba6b-be3ffea2fa4a\") " Nov 02 17:45:12.242134 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.242108 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17d2f69e-081c-420b-bded-9454a7db97fe-kube-api-access-chhdp" pod "17d2f69e-081c-420b-bded-9454a7db97fe" (UID: "17d2f69e-081c-420b-bded-9454a7db97fe"). InnerVolumeSpecName "kube-api-access-chhdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:45:12.242221 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.242145 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a7b68c6-264d-4724-ba6b-be3ffea2fa4a-kube-api-access-lfjmb" pod "4a7b68c6-264d-4724-ba6b-be3ffea2fa4a" (UID: "4a7b68c6-264d-4724-ba6b-be3ffea2fa4a"). InnerVolumeSpecName "kube-api-access-lfjmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:45:12.341486 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.341458 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chhdp\" (UniqueName: \"kubernetes.io/projected/17d2f69e-081c-420b-bded-9454a7db97fe-kube-api-access-chhdp\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:45:12.341486 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.341483 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfjmb\" (UniqueName: \"kubernetes.io/projected/4a7b68c6-264d-4724-ba6b-be3ffea2fa4a-kube-api-access-lfjmb\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:45:12.443293 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.443218 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17d2f69e-081c-420b-bded-9454a7db97fe" path="/var/lib/kubelet/pods/17d2f69e-081c-420b-bded-9454a7db97fe/volumes" Nov 02 17:45:12.443608 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.443594 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a7b68c6-264d-4724-ba6b-be3ffea2fa4a" path="/var/lib/kubelet/pods/4a7b68c6-264d-4724-ba6b-be3ffea2fa4a/volumes" Nov 02 17:45:12.610320 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.610276 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-223215e4-b78cf56ff-kj7gq"] Nov 02 17:45:12.610912 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.610896 4471 cpu_manager.go:414] "containerMap: removing container" podUID="4a7b68c6-264d-4724-ba6b-be3ffea2fa4a" containerName="test-curl-6806-b75dce" Nov 02 17:45:12.610981 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.610913 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a7b68c6-264d-4724-ba6b-be3ffea2fa4a" containerName="test-curl-6806-b75dce" Nov 02 17:45:12.610981 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.610959 4471 cpu_manager.go:414] "containerMap: removing container" podUID="17d2f69e-081c-420b-bded-9454a7db97fe" containerName="test-curl-6803-59249b" Nov 02 17:45:12.610981 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.610964 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17d2f69e-081c-420b-bded-9454a7db97fe" containerName="test-curl-6803-59249b" Nov 02 17:45:12.611075 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.611057 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="17d2f69e-081c-420b-bded-9454a7db97fe" containerName="test-curl-6803-59249b" Nov 02 17:45:12.611075 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.611068 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a7b68c6-264d-4724-ba6b-be3ffea2fa4a" containerName="test-curl-6806-b75dce" Nov 02 17:45:12.614720 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.614700 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-223215e4-b78cf56ff-kj7gq" Nov 02 17:45:12.635108 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.635083 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-223215e4-b78cf56ff-kj7gq"] Nov 02 17:45:12.640929 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.640902 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-223215e4-dockercfg-95gxg\"" Nov 02 17:45:12.640929 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.640925 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-223215e4\"" Nov 02 17:45:12.645167 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.645140 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w56xx\" (UniqueName: \"kubernetes.io/projected/809eb3bd-2daf-40fe-a332-43032c2be57a-kube-api-access-w56xx\") pod \"payload-pre-processing-e2e-ait-223215e4-b78cf56ff-kj7gq\" (UID: \"809eb3bd-2daf-40fe-a332-43032c2be57a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-223215e4-b78cf56ff-kj7gq" Nov 02 17:45:12.645286 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.645193 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/809eb3bd-2daf-40fe-a332-43032c2be57a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-223215e4-b78cf56ff-kj7gq\" (UID: \"809eb3bd-2daf-40fe-a332-43032c2be57a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-223215e4-b78cf56ff-kj7gq" Nov 02 17:45:12.760383 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.757908 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w56xx\" (UniqueName: \"kubernetes.io/projected/809eb3bd-2daf-40fe-a332-43032c2be57a-kube-api-access-w56xx\") pod \"payload-pre-processing-e2e-ait-223215e4-b78cf56ff-kj7gq\" (UID: \"809eb3bd-2daf-40fe-a332-43032c2be57a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-223215e4-b78cf56ff-kj7gq" Nov 02 17:45:12.760383 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.757991 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/809eb3bd-2daf-40fe-a332-43032c2be57a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-223215e4-b78cf56ff-kj7gq\" (UID: \"809eb3bd-2daf-40fe-a332-43032c2be57a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-223215e4-b78cf56ff-kj7gq" Nov 02 17:45:12.760383 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.759207 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/809eb3bd-2daf-40fe-a332-43032c2be57a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-223215e4-b78cf56ff-kj7gq\" (UID: \"809eb3bd-2daf-40fe-a332-43032c2be57a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-223215e4-b78cf56ff-kj7gq" Nov 02 17:45:12.808796 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.808757 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w56xx\" (UniqueName: \"kubernetes.io/projected/809eb3bd-2daf-40fe-a332-43032c2be57a-kube-api-access-w56xx\") pod \"payload-pre-processing-e2e-ait-223215e4-b78cf56ff-kj7gq\" (UID: \"809eb3bd-2daf-40fe-a332-43032c2be57a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-223215e4-b78cf56ff-kj7gq" Nov 02 17:45:12.810548 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.809020 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-223215e4-868c4668c5-8nh97"] Nov 02 17:45:12.902908 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.868520 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-223215e4-868c4668c5-8nh97"] Nov 02 17:45:12.902908 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.868653 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-223215e4-868c4668c5-8nh97" Nov 02 17:45:12.955980 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.955710 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-223215e4-b78cf56ff-kj7gq" Nov 02 17:45:12.985291 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.980657 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2-plugins-config-volume\") pod \"payload-processing-e2e-ait-223215e4-868c4668c5-8nh97\" (UID: \"10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2\") " pod="openshift-ingress/payload-processing-e2e-ait-223215e4-868c4668c5-8nh97" Nov 02 17:45:12.985291 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.980701 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbvlx\" (UniqueName: \"kubernetes.io/projected/10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2-kube-api-access-qbvlx\") pod \"payload-processing-e2e-ait-223215e4-868c4668c5-8nh97\" (UID: \"10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2\") " pod="openshift-ingress/payload-processing-e2e-ait-223215e4-868c4668c5-8nh97" Nov 02 17:45:12.992205 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.989649 4471 generic.go:356] "Generic (PLEG): container finished" podID="4a7b68c6-264d-4724-ba6b-be3ffea2fa4a" containerID="d80a3874a94d69ab02e8a587b4198b8dac2f7c1006b9a2534bb967b17234227a" exitCode=137 Nov 02 17:45:12.992205 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.989822 4471 scope.go:122] "RemoveContainer" containerID="d80a3874a94d69ab02e8a587b4198b8dac2f7c1006b9a2534bb967b17234227a" Nov 02 17:45:12.992205 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:12.989982 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6806-b75dce" Nov 02 17:45:13.045661 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:13.043401 4471 generic.go:356] "Generic (PLEG): container finished" podID="17d2f69e-081c-420b-bded-9454a7db97fe" containerID="688ffa415b9c0456e9a78b592b2cd15ea929743bacdc850205f47a45e50f11b1" exitCode=137 Nov 02 17:45:13.045661 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:13.044693 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6803-59249b" Nov 02 17:45:13.082332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:13.082253 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2-plugins-config-volume\") pod \"payload-processing-e2e-ait-223215e4-868c4668c5-8nh97\" (UID: \"10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2\") " pod="openshift-ingress/payload-processing-e2e-ait-223215e4-868c4668c5-8nh97" Nov 02 17:45:13.083598 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:13.083523 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2-plugins-config-volume\") pod \"payload-processing-e2e-ait-223215e4-868c4668c5-8nh97\" (UID: \"10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2\") " pod="openshift-ingress/payload-processing-e2e-ait-223215e4-868c4668c5-8nh97" Nov 02 17:45:13.087445 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:13.084101 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbvlx\" (UniqueName: \"kubernetes.io/projected/10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2-kube-api-access-qbvlx\") pod \"payload-processing-e2e-ait-223215e4-868c4668c5-8nh97\" (UID: \"10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2\") " pod="openshift-ingress/payload-processing-e2e-ait-223215e4-868c4668c5-8nh97" Nov 02 17:45:13.101354 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:13.101331 4471 scope.go:122] "RemoveContainer" containerID="d80a3874a94d69ab02e8a587b4198b8dac2f7c1006b9a2534bb967b17234227a" Nov 02 17:45:13.101730 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:45:13.101706 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d80a3874a94d69ab02e8a587b4198b8dac2f7c1006b9a2534bb967b17234227a\": container with ID starting with d80a3874a94d69ab02e8a587b4198b8dac2f7c1006b9a2534bb967b17234227a not found: ID does not exist" containerID="d80a3874a94d69ab02e8a587b4198b8dac2f7c1006b9a2534bb967b17234227a" Nov 02 17:45:13.101830 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:13.101743 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d80a3874a94d69ab02e8a587b4198b8dac2f7c1006b9a2534bb967b17234227a"} err="failed to get container status \"d80a3874a94d69ab02e8a587b4198b8dac2f7c1006b9a2534bb967b17234227a\": rpc error: code = NotFound desc = could not find container \"d80a3874a94d69ab02e8a587b4198b8dac2f7c1006b9a2534bb967b17234227a\": container with ID starting with d80a3874a94d69ab02e8a587b4198b8dac2f7c1006b9a2534bb967b17234227a not found: ID does not exist" Nov 02 17:45:13.101830 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:13.101770 4471 scope.go:122] "RemoveContainer" containerID="688ffa415b9c0456e9a78b592b2cd15ea929743bacdc850205f47a45e50f11b1" Nov 02 17:45:13.108548 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:13.108487 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbvlx\" (UniqueName: \"kubernetes.io/projected/10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2-kube-api-access-qbvlx\") pod \"payload-processing-e2e-ait-223215e4-868c4668c5-8nh97\" (UID: \"10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2\") " pod="openshift-ingress/payload-processing-e2e-ait-223215e4-868c4668c5-8nh97" Nov 02 17:45:13.129298 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:13.129164 4471 scope.go:122] "RemoveContainer" containerID="688ffa415b9c0456e9a78b592b2cd15ea929743bacdc850205f47a45e50f11b1" Nov 02 17:45:13.130802 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:45:13.130683 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"688ffa415b9c0456e9a78b592b2cd15ea929743bacdc850205f47a45e50f11b1\": container with ID starting with 688ffa415b9c0456e9a78b592b2cd15ea929743bacdc850205f47a45e50f11b1 not found: ID does not exist" containerID="688ffa415b9c0456e9a78b592b2cd15ea929743bacdc850205f47a45e50f11b1" Nov 02 17:45:13.130802 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:13.130722 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"688ffa415b9c0456e9a78b592b2cd15ea929743bacdc850205f47a45e50f11b1"} err="failed to get container status \"688ffa415b9c0456e9a78b592b2cd15ea929743bacdc850205f47a45e50f11b1\": rpc error: code = NotFound desc = could not find container \"688ffa415b9c0456e9a78b592b2cd15ea929743bacdc850205f47a45e50f11b1\": container with ID starting with 688ffa415b9c0456e9a78b592b2cd15ea929743bacdc850205f47a45e50f11b1 not found: ID does not exist" Nov 02 17:45:13.387792 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:45:13.387756 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod809eb3bd_2daf_40fe_a332_43032c2be57a.slice/crio-5219fc63202660ef28f2feed609bebc9536f137258e7faeb6eb463e8bb91c454 WatchSource:0}: Error finding container 5219fc63202660ef28f2feed609bebc9536f137258e7faeb6eb463e8bb91c454: Status 404 returned error can't find the container with id 5219fc63202660ef28f2feed609bebc9536f137258e7faeb6eb463e8bb91c454 Nov 02 17:45:13.389545 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:13.389519 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-223215e4-b78cf56ff-kj7gq"] Nov 02 17:45:13.408022 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:13.408000 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-223215e4-868c4668c5-8nh97" Nov 02 17:45:13.722409 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:45:13.722378 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10d8c0f7_b825_4d6d_9fd4_e60be9d17ff2.slice/crio-18e0edd1d42081d947f581ab3923cff6db89d3aba00dc8970153080e9ee716d3 WatchSource:0}: Error finding container 18e0edd1d42081d947f581ab3923cff6db89d3aba00dc8970153080e9ee716d3: Status 404 returned error can't find the container with id 18e0edd1d42081d947f581ab3923cff6db89d3aba00dc8970153080e9ee716d3 Nov 02 17:45:13.733469 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:13.733420 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-223215e4-868c4668c5-8nh97"] Nov 02 17:45:14.093329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:14.085560 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-223215e4-b78cf56ff-kj7gq" event={"ID":"809eb3bd-2daf-40fe-a332-43032c2be57a","Type":"ContainerStarted","Data":"b8b628787f008202a574cda0da351bafbc0b7fdc85a42479566531f3e2dc0965"} Nov 02 17:45:14.093329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:14.085658 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-223215e4-b78cf56ff-kj7gq" event={"ID":"809eb3bd-2daf-40fe-a332-43032c2be57a","Type":"ContainerStarted","Data":"5219fc63202660ef28f2feed609bebc9536f137258e7faeb6eb463e8bb91c454"} Nov 02 17:45:14.093329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:14.087092 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-223215e4-b78cf56ff-kj7gq" Nov 02 17:45:14.093329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:14.092007 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-223215e4-868c4668c5-8nh97" event={"ID":"10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2","Type":"ContainerStarted","Data":"81f9001cb3fe13e7bdcba18483c03d0fb5f09c7ff04bbf8a3dab86b082b3de69"} Nov 02 17:45:14.093329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:14.092067 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-223215e4-868c4668c5-8nh97" Nov 02 17:45:14.093329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:14.092095 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-223215e4-868c4668c5-8nh97" event={"ID":"10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2","Type":"ContainerStarted","Data":"18e0edd1d42081d947f581ab3923cff6db89d3aba00dc8970153080e9ee716d3"} Nov 02 17:45:14.142148 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:14.141876 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-223215e4-b78cf56ff-kj7gq" podStartSLOduration=2.141856481 podStartE2EDuration="2.141856481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:14.141453346 +0000 UTC m=+931.836808830" watchObservedRunningTime="2026-11-02 17:45:14.141856481 +0000 UTC m=+931.837211956" Nov 02 17:45:14.188652 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:14.188593 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-223215e4-868c4668c5-8nh97" podStartSLOduration=2.188578053 podStartE2EDuration="2.188578053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:14.186020611 +0000 UTC m=+931.881376104" watchObservedRunningTime="2026-11-02 17:45:14.188578053 +0000 UTC m=+931.883933560" Nov 02 17:45:14.800778 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:14.800744 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-556cd8a1-cb497c9c-bdlhx"] Nov 02 17:45:14.805052 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:14.805035 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-556cd8a1-cb497c9c-bdlhx" Nov 02 17:45:14.820930 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:14.820908 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-556cd8a1\"" Nov 02 17:45:14.830474 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:14.830430 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f5643c-8167-4504-ba33-51b35aa781d7-plugins-config-volume\") pod \"payload-processing-e2e-ait-556cd8a1-cb497c9c-bdlhx\" (UID: \"15f5643c-8167-4504-ba33-51b35aa781d7\") " pod="openshift-ingress/payload-processing-e2e-ait-556cd8a1-cb497c9c-bdlhx" Nov 02 17:45:14.830632 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:14.830520 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-556cd8a1-cb497c9c-bdlhx"] Nov 02 17:45:14.830632 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:14.830578 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4jzl\" (UniqueName: \"kubernetes.io/projected/15f5643c-8167-4504-ba33-51b35aa781d7-kube-api-access-f4jzl\") pod \"payload-processing-e2e-ait-556cd8a1-cb497c9c-bdlhx\" (UID: \"15f5643c-8167-4504-ba33-51b35aa781d7\") " pod="openshift-ingress/payload-processing-e2e-ait-556cd8a1-cb497c9c-bdlhx" Nov 02 17:45:14.873821 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:14.873787 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-556cd8a1-5b4cc7856c-k289v"] Nov 02 17:45:14.879339 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:14.879297 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-556cd8a1-5b4cc7856c-k289v" Nov 02 17:45:14.896536 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:14.896513 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-556cd8a1-dockercfg-4mttc\"" Nov 02 17:45:14.899477 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:14.899447 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-556cd8a1-5b4cc7856c-k289v"] Nov 02 17:45:14.983329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:14.952416 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ad02b4a-be6d-4f85-b8fc-8e9188e35649-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-556cd8a1-5b4cc7856c-k289v\" (UID: \"7ad02b4a-be6d-4f85-b8fc-8e9188e35649\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-556cd8a1-5b4cc7856c-k289v" Nov 02 17:45:14.983654 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:14.983614 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt4xz\" (UniqueName: \"kubernetes.io/projected/7ad02b4a-be6d-4f85-b8fc-8e9188e35649-kube-api-access-vt4xz\") pod \"payload-pre-processing-e2e-ait-556cd8a1-5b4cc7856c-k289v\" (UID: \"7ad02b4a-be6d-4f85-b8fc-8e9188e35649\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-556cd8a1-5b4cc7856c-k289v" Nov 02 17:45:15.008335 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:14.983909 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f5643c-8167-4504-ba33-51b35aa781d7-plugins-config-volume\") pod \"payload-processing-e2e-ait-556cd8a1-cb497c9c-bdlhx\" (UID: \"15f5643c-8167-4504-ba33-51b35aa781d7\") " pod="openshift-ingress/payload-processing-e2e-ait-556cd8a1-cb497c9c-bdlhx" Nov 02 17:45:15.082638 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:15.082550 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4jzl\" (UniqueName: \"kubernetes.io/projected/15f5643c-8167-4504-ba33-51b35aa781d7-kube-api-access-f4jzl\") pod \"payload-processing-e2e-ait-556cd8a1-cb497c9c-bdlhx\" (UID: \"15f5643c-8167-4504-ba33-51b35aa781d7\") " pod="openshift-ingress/payload-processing-e2e-ait-556cd8a1-cb497c9c-bdlhx" Nov 02 17:45:15.095385 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:14.999252 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f5643c-8167-4504-ba33-51b35aa781d7-plugins-config-volume\") pod \"payload-processing-e2e-ait-556cd8a1-cb497c9c-bdlhx\" (UID: \"15f5643c-8167-4504-ba33-51b35aa781d7\") " pod="openshift-ingress/payload-processing-e2e-ait-556cd8a1-cb497c9c-bdlhx" Nov 02 17:45:15.146358 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:15.145098 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4jzl\" (UniqueName: \"kubernetes.io/projected/15f5643c-8167-4504-ba33-51b35aa781d7-kube-api-access-f4jzl\") pod \"payload-processing-e2e-ait-556cd8a1-cb497c9c-bdlhx\" (UID: \"15f5643c-8167-4504-ba33-51b35aa781d7\") " pod="openshift-ingress/payload-processing-e2e-ait-556cd8a1-cb497c9c-bdlhx" Nov 02 17:45:15.195326 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:15.189731 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ad02b4a-be6d-4f85-b8fc-8e9188e35649-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-556cd8a1-5b4cc7856c-k289v\" (UID: \"7ad02b4a-be6d-4f85-b8fc-8e9188e35649\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-556cd8a1-5b4cc7856c-k289v" Nov 02 17:45:15.195326 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:15.189807 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt4xz\" (UniqueName: \"kubernetes.io/projected/7ad02b4a-be6d-4f85-b8fc-8e9188e35649-kube-api-access-vt4xz\") pod \"payload-pre-processing-e2e-ait-556cd8a1-5b4cc7856c-k289v\" (UID: \"7ad02b4a-be6d-4f85-b8fc-8e9188e35649\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-556cd8a1-5b4cc7856c-k289v" Nov 02 17:45:15.195326 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:15.192196 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ad02b4a-be6d-4f85-b8fc-8e9188e35649-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-556cd8a1-5b4cc7856c-k289v\" (UID: \"7ad02b4a-be6d-4f85-b8fc-8e9188e35649\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-556cd8a1-5b4cc7856c-k289v" Nov 02 17:45:15.258833 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:15.243526 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt4xz\" (UniqueName: \"kubernetes.io/projected/7ad02b4a-be6d-4f85-b8fc-8e9188e35649-kube-api-access-vt4xz\") pod \"payload-pre-processing-e2e-ait-556cd8a1-5b4cc7856c-k289v\" (UID: \"7ad02b4a-be6d-4f85-b8fc-8e9188e35649\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-556cd8a1-5b4cc7856c-k289v" Nov 02 17:45:15.419466 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:15.419391 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-556cd8a1-cb497c9c-bdlhx" Nov 02 17:45:15.493115 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:15.493070 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-556cd8a1-5b4cc7856c-k289v" Nov 02 17:45:15.718019 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:45:15.717993 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15f5643c_8167_4504_ba33_51b35aa781d7.slice/crio-7670dbcbf41b890702554ceee55ee90255a3ec61d5dcf5f4ff551621214fba48 WatchSource:0}: Error finding container 7670dbcbf41b890702554ceee55ee90255a3ec61d5dcf5f4ff551621214fba48: Status 404 returned error can't find the container with id 7670dbcbf41b890702554ceee55ee90255a3ec61d5dcf5f4ff551621214fba48 Nov 02 17:45:15.719722 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:15.719700 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-556cd8a1-cb497c9c-bdlhx"] Nov 02 17:45:15.800301 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:45:15.800275 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad02b4a_be6d_4f85_b8fc_8e9188e35649.slice/crio-643862ed48eb5c34b0106a9bb4ac4bbc3e4ffa71616f8efbb04953d359945bdb WatchSource:0}: Error finding container 643862ed48eb5c34b0106a9bb4ac4bbc3e4ffa71616f8efbb04953d359945bdb: Status 404 returned error can't find the container with id 643862ed48eb5c34b0106a9bb4ac4bbc3e4ffa71616f8efbb04953d359945bdb Nov 02 17:45:15.803720 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:15.803692 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-556cd8a1-5b4cc7856c-k289v"] Nov 02 17:45:16.143665 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:16.143626 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-556cd8a1-5b4cc7856c-k289v" event={"ID":"7ad02b4a-be6d-4f85-b8fc-8e9188e35649","Type":"ContainerStarted","Data":"7138748a51e3bb067098d3a466cba592fe020a1ef9adb661d92451dec4c85def"} Nov 02 17:45:16.143841 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:16.143684 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-556cd8a1-5b4cc7856c-k289v" event={"ID":"7ad02b4a-be6d-4f85-b8fc-8e9188e35649","Type":"ContainerStarted","Data":"643862ed48eb5c34b0106a9bb4ac4bbc3e4ffa71616f8efbb04953d359945bdb"} Nov 02 17:45:16.143841 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:16.143715 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-556cd8a1-5b4cc7856c-k289v" Nov 02 17:45:16.145373 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:16.145337 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-556cd8a1-cb497c9c-bdlhx" event={"ID":"15f5643c-8167-4504-ba33-51b35aa781d7","Type":"ContainerStarted","Data":"cee3288066dbc33c1586e32b762cb1b55d82a33839852109183a4a56083f8d2d"} Nov 02 17:45:16.145495 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:16.145391 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-556cd8a1-cb497c9c-bdlhx" event={"ID":"15f5643c-8167-4504-ba33-51b35aa781d7","Type":"ContainerStarted","Data":"7670dbcbf41b890702554ceee55ee90255a3ec61d5dcf5f4ff551621214fba48"} Nov 02 17:45:16.211285 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:16.211216 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-556cd8a1-5b4cc7856c-k289v" podStartSLOduration=2.211192355 podStartE2EDuration="2.211192355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:16.200909794 +0000 UTC m=+933.896265329" watchObservedRunningTime="2026-11-02 17:45:16.211192355 +0000 UTC m=+933.906547828" Nov 02 17:45:16.299424 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:16.299326 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-556cd8a1-cb497c9c-bdlhx" podStartSLOduration=2.2992876 podStartE2EDuration="2.2992876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:16.2642935 +0000 UTC m=+933.959649028" watchObservedRunningTime="2026-11-02 17:45:16.2992876 +0000 UTC m=+933.994643074" Nov 02 17:45:17.152216 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:17.152188 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-556cd8a1-cb497c9c-bdlhx" Nov 02 17:45:20.947148 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:20.947092 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:21.239589 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:21.239480 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:21.370514 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:21.370461 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:21.388338 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:21.372561 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 02 17:45:21.388338 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:21.372884 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 02 17:45:21.388338 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:21.372939 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:21.474370 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:21.438431 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:21.964334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:21.963410 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:22.007385 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:21.964962 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:45:22.249498 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:22.249396 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:22.251060 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:22.251032 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:45:22.376338 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:22.372889 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:22.416335 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:22.395047 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:22.454400 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:22.413060 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:45:22.454697 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:22.454672 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 17:45:22.504739 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:22.376413 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:22.504739 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:22.444801 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:22.505781 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:22.413576 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:22.737329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:22.730141 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:45:22.737329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:22.730231 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:45:22.737329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:22.730270 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:45:23.023455 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:23.023403 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:23.324393 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:23.320468 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:23.436468 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:23.436420 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:23.462791 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:23.462745 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": context deadline exceeded" Nov 02 17:45:23.523330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:23.502672 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:23.523694 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:23.523650 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:23.524195 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:23.524167 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 02 17:45:23.716451 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:23.715409 4471 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:45:23.716451 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:23.715482 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" podUID="006cd169-c635-4f97-a8d1-c21aa6c077c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:24.048879 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:24.048827 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:24.146872 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:24.146828 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:45:24.147063 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:24.146899 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" podUID="636cf7f7-ba1a-4f54-9db0-6d8ad2909424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:24.349524 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:24.342910 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:24.522331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:24.513971 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:24.534432 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:24.534381 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:45:24.561333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:24.541447 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:24.561333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:24.541523 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 02 17:45:24.561333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:24.541563 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 02 17:45:24.974432 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:24.971938 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:25.034675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:25.034641 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:45:25.035008 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:25.034989 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:45:25.053416 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:25.053373 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:25.077435 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:25.077400 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 17:45:25.078115 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:25.078094 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:45:25.159461 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:25.159428 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-223215e4-868c4668c5-8nh97" Nov 02 17:45:25.459472 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:25.458355 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:45:25.551323 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:25.533774 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:45:25.551323 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:25.535575 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:45:25.701504 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:25.701469 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-30a972ba-899549479-jv7r7"] Nov 02 17:45:25.707255 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:25.707230 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-30a972ba-899549479-jv7r7" Nov 02 17:45:25.725167 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:25.725098 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-30a972ba-899549479-jv7r7"] Nov 02 17:45:25.726725 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:25.725844 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-30a972ba-dockercfg-vrfxl\"" Nov 02 17:45:25.726725 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:25.725847 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-30a972ba\"" Nov 02 17:45:25.761686 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:25.761652 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b4ac904-c840-4caa-847b-e77250eaef2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-30a972ba-899549479-jv7r7\" (UID: \"3b4ac904-c840-4caa-847b-e77250eaef2d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-30a972ba-899549479-jv7r7" Nov 02 17:45:25.761855 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:25.761692 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpphk\" (UniqueName: \"kubernetes.io/projected/3b4ac904-c840-4caa-847b-e77250eaef2d-kube-api-access-kpphk\") pod \"payload-pre-processing-e2e-ait-30a972ba-899549479-jv7r7\" (UID: \"3b4ac904-c840-4caa-847b-e77250eaef2d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-30a972ba-899549479-jv7r7" Nov 02 17:45:25.772781 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:25.772751 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-30a972ba-7867f88c69-7cbk4"] Nov 02 17:45:25.781917 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:25.780582 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-30a972ba-7867f88c69-7cbk4" Nov 02 17:45:25.888160 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:25.888115 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpn4s\" (UniqueName: \"kubernetes.io/projected/5ea7040b-14c0-4b5c-ad13-0a77e8026bd9-kube-api-access-dpn4s\") pod \"payload-processing-e2e-ait-30a972ba-7867f88c69-7cbk4\" (UID: \"5ea7040b-14c0-4b5c-ad13-0a77e8026bd9\") " pod="openshift-ingress/payload-processing-e2e-ait-30a972ba-7867f88c69-7cbk4" Nov 02 17:45:25.888686 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:25.888653 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b4ac904-c840-4caa-847b-e77250eaef2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-30a972ba-899549479-jv7r7\" (UID: \"3b4ac904-c840-4caa-847b-e77250eaef2d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-30a972ba-899549479-jv7r7" Nov 02 17:45:25.892436 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:25.892409 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpphk\" (UniqueName: \"kubernetes.io/projected/3b4ac904-c840-4caa-847b-e77250eaef2d-kube-api-access-kpphk\") pod \"payload-pre-processing-e2e-ait-30a972ba-899549479-jv7r7\" (UID: \"3b4ac904-c840-4caa-847b-e77250eaef2d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-30a972ba-899549479-jv7r7" Nov 02 17:45:25.947348 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:25.899575 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ea7040b-14c0-4b5c-ad13-0a77e8026bd9-plugins-config-volume\") pod \"payload-processing-e2e-ait-30a972ba-7867f88c69-7cbk4\" (UID: \"5ea7040b-14c0-4b5c-ad13-0a77e8026bd9\") " pod="openshift-ingress/payload-processing-e2e-ait-30a972ba-7867f88c69-7cbk4" Nov 02 17:45:25.947348 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:25.900915 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b4ac904-c840-4caa-847b-e77250eaef2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-30a972ba-899549479-jv7r7\" (UID: \"3b4ac904-c840-4caa-847b-e77250eaef2d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-30a972ba-899549479-jv7r7" Nov 02 17:45:26.034333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:26.016241 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-30a972ba-7867f88c69-7cbk4"] Nov 02 17:45:26.034333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:26.017453 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ea7040b-14c0-4b5c-ad13-0a77e8026bd9-plugins-config-volume\") pod \"payload-processing-e2e-ait-30a972ba-7867f88c69-7cbk4\" (UID: \"5ea7040b-14c0-4b5c-ad13-0a77e8026bd9\") " pod="openshift-ingress/payload-processing-e2e-ait-30a972ba-7867f88c69-7cbk4" Nov 02 17:45:26.034333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:26.017682 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpn4s\" (UniqueName: \"kubernetes.io/projected/5ea7040b-14c0-4b5c-ad13-0a77e8026bd9-kube-api-access-dpn4s\") pod \"payload-processing-e2e-ait-30a972ba-7867f88c69-7cbk4\" (UID: \"5ea7040b-14c0-4b5c-ad13-0a77e8026bd9\") " pod="openshift-ingress/payload-processing-e2e-ait-30a972ba-7867f88c69-7cbk4" Nov 02 17:45:26.071456 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:26.035143 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ea7040b-14c0-4b5c-ad13-0a77e8026bd9-plugins-config-volume\") pod \"payload-processing-e2e-ait-30a972ba-7867f88c69-7cbk4\" (UID: \"5ea7040b-14c0-4b5c-ad13-0a77e8026bd9\") " pod="openshift-ingress/payload-processing-e2e-ait-30a972ba-7867f88c69-7cbk4" Nov 02 17:45:26.151418 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:26.149252 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpphk\" (UniqueName: \"kubernetes.io/projected/3b4ac904-c840-4caa-847b-e77250eaef2d-kube-api-access-kpphk\") pod \"payload-pre-processing-e2e-ait-30a972ba-899549479-jv7r7\" (UID: \"3b4ac904-c840-4caa-847b-e77250eaef2d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-30a972ba-899549479-jv7r7" Nov 02 17:45:26.170836 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:26.154952 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpn4s\" (UniqueName: \"kubernetes.io/projected/5ea7040b-14c0-4b5c-ad13-0a77e8026bd9-kube-api-access-dpn4s\") pod \"payload-processing-e2e-ait-30a972ba-7867f88c69-7cbk4\" (UID: \"5ea7040b-14c0-4b5c-ad13-0a77e8026bd9\") " pod="openshift-ingress/payload-processing-e2e-ait-30a972ba-7867f88c69-7cbk4" Nov 02 17:45:26.176365 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:26.176334 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-223215e4-b78cf56ff-kj7gq" Nov 02 17:45:26.323178 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:26.323099 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-30a972ba-899549479-jv7r7" Nov 02 17:45:26.432070 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:26.432021 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-30a972ba-7867f88c69-7cbk4" Nov 02 17:45:26.734883 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:26.720964 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-30a972ba-899549479-jv7r7"] Nov 02 17:45:26.777891 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:26.777863 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-30a972ba-7867f88c69-7cbk4"] Nov 02 17:45:26.782587 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:45:26.782560 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea7040b_14c0_4b5c_ad13_0a77e8026bd9.slice/crio-5cb4e3d0afc1784cc8fad42c69ceebe0f55c69838b7ce9e9751ee14f18d22f9f WatchSource:0}: Error finding container 5cb4e3d0afc1784cc8fad42c69ceebe0f55c69838b7ce9e9751ee14f18d22f9f: Status 404 returned error can't find the container with id 5cb4e3d0afc1784cc8fad42c69ceebe0f55c69838b7ce9e9751ee14f18d22f9f Nov 02 17:45:27.008928 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:27.008830 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-30a972ba-7867f88c69-7cbk4" event={"ID":"5ea7040b-14c0-4b5c-ad13-0a77e8026bd9","Type":"ContainerStarted","Data":"c32a060e1edd098a5db29e495577d232ea840f4b6315465d0f542abdf1f84e9a"} Nov 02 17:45:27.008928 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:27.008901 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-30a972ba-7867f88c69-7cbk4" event={"ID":"5ea7040b-14c0-4b5c-ad13-0a77e8026bd9","Type":"ContainerStarted","Data":"5cb4e3d0afc1784cc8fad42c69ceebe0f55c69838b7ce9e9751ee14f18d22f9f"} Nov 02 17:45:27.009160 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:27.009054 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-30a972ba-7867f88c69-7cbk4" Nov 02 17:45:27.011047 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:27.011020 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-30a972ba-899549479-jv7r7" event={"ID":"3b4ac904-c840-4caa-847b-e77250eaef2d","Type":"ContainerStarted","Data":"ff7b58bf9b813b87cb4e588c18a7583c6a781687f4f28fbeaf7ac90de8482c6d"} Nov 02 17:45:27.011157 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:27.011076 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-30a972ba-899549479-jv7r7" Nov 02 17:45:27.011157 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:27.011103 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-30a972ba-899549479-jv7r7" event={"ID":"3b4ac904-c840-4caa-847b-e77250eaef2d","Type":"ContainerStarted","Data":"fc15e911a78da8aec81db6b95d71af16fe82d8359a704d78f86e181f618b19e7"} Nov 02 17:45:27.064756 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:27.064707 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-30a972ba-7867f88c69-7cbk4" podStartSLOduration=2.064685214 podStartE2EDuration="2.064685214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:27.063819642 +0000 UTC m=+944.759175117" watchObservedRunningTime="2026-11-02 17:45:27.064685214 +0000 UTC m=+944.760040696" Nov 02 17:45:27.110358 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:27.110282 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-30a972ba-899549479-jv7r7" podStartSLOduration=2.110267509 podStartE2EDuration="2.110267509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:27.106929926 +0000 UTC m=+944.802285419" watchObservedRunningTime="2026-11-02 17:45:27.110267509 +0000 UTC m=+944.805622981" Nov 02 17:45:27.155242 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:27.155211 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-556cd8a1-5b4cc7856c-k289v" Nov 02 17:45:28.162290 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:28.162257 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-556cd8a1-cb497c9c-bdlhx" Nov 02 17:45:30.286071 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:30.286039 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-223215e4-59469596d7-2lmwt"] Nov 02 17:45:30.290741 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:30.290715 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-223215e4-59469596d7-2lmwt" Nov 02 17:45:30.321672 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:30.321645 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-223215e4-serving-cert\"" Nov 02 17:45:30.326638 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:30.326605 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-223215e4-59469596d7-2lmwt"] Nov 02 17:45:30.409179 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:30.409133 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8278f2e9-f3b2-4609-a4a2-6db72c7647a4-maas-api-tls\") pod \"maas-api-e2e-ait-223215e4-59469596d7-2lmwt\" (UID: \"8278f2e9-f3b2-4609-a4a2-6db72c7647a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-223215e4-59469596d7-2lmwt" Nov 02 17:45:30.409179 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:30.409168 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8278f2e9-f3b2-4609-a4a2-6db72c7647a4-metrics-certs\") pod \"maas-api-e2e-ait-223215e4-59469596d7-2lmwt\" (UID: \"8278f2e9-f3b2-4609-a4a2-6db72c7647a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-223215e4-59469596d7-2lmwt" Nov 02 17:45:30.409439 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:30.409358 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wzhs\" (UniqueName: \"kubernetes.io/projected/8278f2e9-f3b2-4609-a4a2-6db72c7647a4-kube-api-access-8wzhs\") pod \"maas-api-e2e-ait-223215e4-59469596d7-2lmwt\" (UID: \"8278f2e9-f3b2-4609-a4a2-6db72c7647a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-223215e4-59469596d7-2lmwt" Nov 02 17:45:30.409439 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:30.409405 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8278f2e9-f3b2-4609-a4a2-6db72c7647a4-openshift-service-ca\") pod \"maas-api-e2e-ait-223215e4-59469596d7-2lmwt\" (UID: \"8278f2e9-f3b2-4609-a4a2-6db72c7647a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-223215e4-59469596d7-2lmwt" Nov 02 17:45:30.554411 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:30.539509 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wzhs\" (UniqueName: \"kubernetes.io/projected/8278f2e9-f3b2-4609-a4a2-6db72c7647a4-kube-api-access-8wzhs\") pod \"maas-api-e2e-ait-223215e4-59469596d7-2lmwt\" (UID: \"8278f2e9-f3b2-4609-a4a2-6db72c7647a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-223215e4-59469596d7-2lmwt" Nov 02 17:45:30.554863 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:30.554822 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8278f2e9-f3b2-4609-a4a2-6db72c7647a4-openshift-service-ca\") pod \"maas-api-e2e-ait-223215e4-59469596d7-2lmwt\" (UID: \"8278f2e9-f3b2-4609-a4a2-6db72c7647a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-223215e4-59469596d7-2lmwt" Nov 02 17:45:30.555547 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:30.555520 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8278f2e9-f3b2-4609-a4a2-6db72c7647a4-maas-api-tls\") pod \"maas-api-e2e-ait-223215e4-59469596d7-2lmwt\" (UID: \"8278f2e9-f3b2-4609-a4a2-6db72c7647a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-223215e4-59469596d7-2lmwt" Nov 02 17:45:30.565328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:30.555681 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8278f2e9-f3b2-4609-a4a2-6db72c7647a4-metrics-certs\") pod \"maas-api-e2e-ait-223215e4-59469596d7-2lmwt\" (UID: \"8278f2e9-f3b2-4609-a4a2-6db72c7647a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-223215e4-59469596d7-2lmwt" Nov 02 17:45:30.595771 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:30.568907 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8278f2e9-f3b2-4609-a4a2-6db72c7647a4-openshift-service-ca\") pod \"maas-api-e2e-ait-223215e4-59469596d7-2lmwt\" (UID: \"8278f2e9-f3b2-4609-a4a2-6db72c7647a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-223215e4-59469596d7-2lmwt" Nov 02 17:45:30.675554 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:30.675525 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wzhs\" (UniqueName: \"kubernetes.io/projected/8278f2e9-f3b2-4609-a4a2-6db72c7647a4-kube-api-access-8wzhs\") pod \"maas-api-e2e-ait-223215e4-59469596d7-2lmwt\" (UID: \"8278f2e9-f3b2-4609-a4a2-6db72c7647a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-223215e4-59469596d7-2lmwt" Nov 02 17:45:30.687930 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:30.687895 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8278f2e9-f3b2-4609-a4a2-6db72c7647a4-metrics-certs\") pod \"maas-api-e2e-ait-223215e4-59469596d7-2lmwt\" (UID: \"8278f2e9-f3b2-4609-a4a2-6db72c7647a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-223215e4-59469596d7-2lmwt" Nov 02 17:45:30.699075 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:30.699040 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8278f2e9-f3b2-4609-a4a2-6db72c7647a4-maas-api-tls\") pod \"maas-api-e2e-ait-223215e4-59469596d7-2lmwt\" (UID: \"8278f2e9-f3b2-4609-a4a2-6db72c7647a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-223215e4-59469596d7-2lmwt" Nov 02 17:45:30.908699 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:30.908604 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-223215e4-59469596d7-2lmwt" Nov 02 17:45:31.208982 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:45:31.208948 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8278f2e9_f3b2_4609_a4a2_6db72c7647a4.slice/crio-f082b2de3450cae68738c3f888fce0e0ffc3c45323735078d4276b12a05b2b22 WatchSource:0}: Error finding container f082b2de3450cae68738c3f888fce0e0ffc3c45323735078d4276b12a05b2b22: Status 404 returned error can't find the container with id f082b2de3450cae68738c3f888fce0e0ffc3c45323735078d4276b12a05b2b22 Nov 02 17:45:31.210968 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:31.210929 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-223215e4-59469596d7-2lmwt"] Nov 02 17:45:32.060176 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:32.060073 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-223215e4-59469596d7-2lmwt" event={"ID":"8278f2e9-f3b2-4609-a4a2-6db72c7647a4","Type":"ContainerStarted","Data":"bca336d5bd0b204854cc5b7b37bc2771ea795bf397109cf3ccbb7bc10a952f14"} Nov 02 17:45:32.060176 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:32.060137 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-223215e4-59469596d7-2lmwt" event={"ID":"8278f2e9-f3b2-4609-a4a2-6db72c7647a4","Type":"ContainerStarted","Data":"f082b2de3450cae68738c3f888fce0e0ffc3c45323735078d4276b12a05b2b22"} Nov 02 17:45:32.060748 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:32.060447 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-223215e4-59469596d7-2lmwt" Nov 02 17:45:32.109357 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:32.109247 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-223215e4-59469596d7-2lmwt" podStartSLOduration=1.471094897 podStartE2EDuration="2.109230797s" totalImagesPullingTime="638.1359ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:32.107445325 +0000 UTC m=+949.802800799" watchObservedRunningTime="2026-11-02 17:45:32.109230797 +0000 UTC m=+949.804586269" Nov 02 17:45:38.020508 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:38.020476 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-30a972ba-7867f88c69-7cbk4" Nov 02 17:45:38.021073 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:38.020552 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-30a972ba-899549479-jv7r7" Nov 02 17:45:38.076056 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:38.076027 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-223215e4-59469596d7-2lmwt" Nov 02 17:45:39.184052 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:45:39.184012 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:45:43.344081 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:43.344046 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2"] Nov 02 17:45:43.349827 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:43.349805 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2" Nov 02 17:45:43.372455 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:43.372422 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2"] Nov 02 17:45:43.379239 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:43.379213 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-30a972ba-serving-cert\"" Nov 02 17:45:43.488212 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:43.488173 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e820b9-991c-4aaa-bdc3-1ba1076b5e81-maas-api-tls\") pod \"maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2\" (UID: \"16e820b9-991c-4aaa-bdc3-1ba1076b5e81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2" Nov 02 17:45:43.488425 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:43.488248 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdhcv\" (UniqueName: \"kubernetes.io/projected/16e820b9-991c-4aaa-bdc3-1ba1076b5e81-kube-api-access-xdhcv\") pod \"maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2\" (UID: \"16e820b9-991c-4aaa-bdc3-1ba1076b5e81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2" Nov 02 17:45:43.488425 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:43.488414 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e820b9-991c-4aaa-bdc3-1ba1076b5e81-metrics-certs\") pod \"maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2\" (UID: \"16e820b9-991c-4aaa-bdc3-1ba1076b5e81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2" Nov 02 17:45:43.488619 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:43.488599 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e820b9-991c-4aaa-bdc3-1ba1076b5e81-openshift-service-ca\") pod \"maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2\" (UID: \"16e820b9-991c-4aaa-bdc3-1ba1076b5e81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2" Nov 02 17:45:43.590472 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:43.590430 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdhcv\" (UniqueName: \"kubernetes.io/projected/16e820b9-991c-4aaa-bdc3-1ba1076b5e81-kube-api-access-xdhcv\") pod \"maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2\" (UID: \"16e820b9-991c-4aaa-bdc3-1ba1076b5e81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2" Nov 02 17:45:43.590697 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:43.590553 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e820b9-991c-4aaa-bdc3-1ba1076b5e81-metrics-certs\") pod \"maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2\" (UID: \"16e820b9-991c-4aaa-bdc3-1ba1076b5e81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2" Nov 02 17:45:43.590918 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:43.590815 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e820b9-991c-4aaa-bdc3-1ba1076b5e81-openshift-service-ca\") pod \"maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2\" (UID: \"16e820b9-991c-4aaa-bdc3-1ba1076b5e81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2" Nov 02 17:45:43.591069 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:43.590974 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e820b9-991c-4aaa-bdc3-1ba1076b5e81-maas-api-tls\") pod \"maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2\" (UID: \"16e820b9-991c-4aaa-bdc3-1ba1076b5e81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2" Nov 02 17:45:43.595755 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:43.592130 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e820b9-991c-4aaa-bdc3-1ba1076b5e81-openshift-service-ca\") pod \"maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2\" (UID: \"16e820b9-991c-4aaa-bdc3-1ba1076b5e81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2" Nov 02 17:45:43.602928 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:43.600845 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e820b9-991c-4aaa-bdc3-1ba1076b5e81-metrics-certs\") pod \"maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2\" (UID: \"16e820b9-991c-4aaa-bdc3-1ba1076b5e81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2" Nov 02 17:45:43.602928 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:43.601257 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e820b9-991c-4aaa-bdc3-1ba1076b5e81-maas-api-tls\") pod \"maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2\" (UID: \"16e820b9-991c-4aaa-bdc3-1ba1076b5e81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2" Nov 02 17:45:43.628937 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:43.627452 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdhcv\" (UniqueName: \"kubernetes.io/projected/16e820b9-991c-4aaa-bdc3-1ba1076b5e81-kube-api-access-xdhcv\") pod \"maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2\" (UID: \"16e820b9-991c-4aaa-bdc3-1ba1076b5e81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2" Nov 02 17:45:43.669221 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:43.668606 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2" Nov 02 17:45:44.033127 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:45:44.033098 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e820b9_991c_4aaa_bdc3_1ba1076b5e81.slice/crio-cac6b27b132d2a81f224a1d5724cb3daa820097b39155781fa5a30020e411cf2 WatchSource:0}: Error finding container cac6b27b132d2a81f224a1d5724cb3daa820097b39155781fa5a30020e411cf2: Status 404 returned error can't find the container with id cac6b27b132d2a81f224a1d5724cb3daa820097b39155781fa5a30020e411cf2 Nov 02 17:45:44.035131 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:44.035098 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2"] Nov 02 17:45:44.161395 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:44.161357 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2" event={"ID":"16e820b9-991c-4aaa-bdc3-1ba1076b5e81","Type":"ContainerStarted","Data":"cac6b27b132d2a81f224a1d5724cb3daa820097b39155781fa5a30020e411cf2"} Nov 02 17:45:45.170498 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:45.170416 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2" event={"ID":"16e820b9-991c-4aaa-bdc3-1ba1076b5e81","Type":"ContainerStarted","Data":"b3ca280860c6aea185d00c7c8ac04d2967ba3cef32b71539bff266febdc0089c"} Nov 02 17:45:45.170856 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:45.170601 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2" Nov 02 17:45:45.215993 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:45.215941 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2" podStartSLOduration=1.815417187 podStartE2EDuration="2.215927229s" totalImagesPullingTime="400.510042ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:45.214810503 +0000 UTC m=+962.910166000" watchObservedRunningTime="2026-11-02 17:45:45.215927229 +0000 UTC m=+962.911282699" Nov 02 17:45:48.658776 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:48.658743 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:45:51.228335 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:45:51.224471 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2" Nov 02 17:46:01.474278 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:01.474232 4471 generic.go:356] "Generic (PLEG): container finished" podID="1c7aaa66-3d4a-4462-82d4-faf4ed4345f9" containerID="21a78147746a371b82dbfd9c3e52c5e48cab2a88aa7df734c4de840afc6c2022" exitCode=0 Nov 02 17:46:01.474756 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:01.474318 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" event={"ID":"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9","Type":"ContainerDied","Data":"21a78147746a371b82dbfd9c3e52c5e48cab2a88aa7df734c4de840afc6c2022"} Nov 02 17:46:01.474814 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:01.474763 4471 scope.go:122] "RemoveContainer" containerID="21a78147746a371b82dbfd9c3e52c5e48cab2a88aa7df734c4de840afc6c2022" Nov 02 17:46:01.476369 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:01.476341 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:46:01.476433 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:01.476411 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:46:01.476469 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:01.476446 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:46:02.491148 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:02.491112 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" event={"ID":"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9","Type":"ContainerStarted","Data":"49c69119eb64eb39944e116cfaadca086eacb6203c20f4bb605867180932ce2b"} Nov 02 17:46:03.242583 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:03.242552 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:46:03.242747 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:03.242707 4471 patch_prober.go:29] interesting pod/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 17:46:03.242791 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:03.242758 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" podUID="1c7aaa66-3d4a-4462-82d4-faf4ed4345f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 17:46:04.242374 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:04.242336 4471 patch_prober.go:29] interesting pod/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 17:46:04.242881 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:04.242400 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" podUID="1c7aaa66-3d4a-4462-82d4-faf4ed4345f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 17:46:04.940618 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:04.940587 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw"] Nov 02 17:46:04.970357 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:04.963585 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 17:46:04.990621 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:04.990589 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw"] Nov 02 17:46:05.104640 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:05.104598 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-7l8nw\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 17:46:05.104833 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:05.104729 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-7l8nw\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 17:46:05.104833 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:05.104808 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-7l8nw\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 17:46:05.104950 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:05.104872 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-7l8nw\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 17:46:05.104950 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:05.104904 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-7l8nw\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 17:46:05.105052 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:05.104971 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-7l8nw\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 17:46:05.105132 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:05.105111 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6cv2\" (UniqueName: \"kubernetes.io/projected/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-kube-api-access-p6cv2\") pod \"maas-default-gateway-openshift-default-76f4876667-7l8nw\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 17:46:05.105451 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:05.105425 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-7l8nw\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 17:46:05.106068 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:05.106036 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-7l8nw\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 17:46:05.207925 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:05.207835 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-7l8nw\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 17:46:05.207925 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:05.207919 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-7l8nw\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 17:46:05.208161 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:05.207950 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-7l8nw\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 17:46:05.208161 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:05.207993 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-7l8nw\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 17:46:05.208161 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:05.208015 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-7l8nw\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 17:46:05.208161 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:05.208063 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-7l8nw\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 17:46:05.208436 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:05.208176 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6cv2\" (UniqueName: \"kubernetes.io/projected/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-kube-api-access-p6cv2\") pod \"maas-default-gateway-openshift-default-76f4876667-7l8nw\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 17:46:05.208436 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:05.208209 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-7l8nw\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 17:46:05.208436 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:05.208226 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-7l8nw\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 17:46:05.208580 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:05.208453 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-7l8nw\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 17:46:05.208580 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:05.208478 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-7l8nw\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 17:46:05.208776 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:05.208750 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-7l8nw\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 17:46:05.208816 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:05.208778 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-7l8nw\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 17:46:05.208816 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:05.208753 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-7l8nw\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 17:46:05.210646 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:05.210623 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-7l8nw\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 17:46:05.210771 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:05.210747 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-7l8nw\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 17:46:05.234624 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:05.234596 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-7l8nw\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 17:46:05.234861 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:05.234840 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6cv2\" (UniqueName: \"kubernetes.io/projected/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-kube-api-access-p6cv2\") pod \"maas-default-gateway-openshift-default-76f4876667-7l8nw\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 17:46:05.241777 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:05.241756 4471 patch_prober.go:29] interesting pod/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 17:46:05.241883 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:05.241797 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" podUID="1c7aaa66-3d4a-4462-82d4-faf4ed4345f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 17:46:05.414209 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:05.414173 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 17:46:05.785265 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:05.785231 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw"] Nov 02 17:46:05.786564 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:46:05.786535 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa74f62_fe2f_4238_9f58_4a0a20f7892a.slice/crio-598d8ee7d0693ef2f0eccd864f5851fb1491644ace2eb4444e12c46c2a246aab WatchSource:0}: Error finding container 598d8ee7d0693ef2f0eccd864f5851fb1491644ace2eb4444e12c46c2a246aab: Status 404 returned error can't find the container with id 598d8ee7d0693ef2f0eccd864f5851fb1491644ace2eb4444e12c46c2a246aab Nov 02 17:46:05.789681 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:05.789633 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:46:05.789790 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:05.789717 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:46:05.789864 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:05.789784 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:46:06.242400 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:06.242299 4471 patch_prober.go:29] interesting pod/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 17:46:06.242400 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:06.242361 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" podUID="1c7aaa66-3d4a-4462-82d4-faf4ed4345f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 17:46:06.651065 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:06.651023 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" event={"ID":"1aa74f62-fe2f-4238-9f58-4a0a20f7892a","Type":"ContainerStarted","Data":"23a9aaf637c69775424d91c36553ff299b37381bff46db3b401612c0853ec643"} Nov 02 17:46:06.651689 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:06.651662 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" event={"ID":"1aa74f62-fe2f-4238-9f58-4a0a20f7892a","Type":"ContainerStarted","Data":"598d8ee7d0693ef2f0eccd864f5851fb1491644ace2eb4444e12c46c2a246aab"} Nov 02 17:46:06.746453 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:06.746377 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" podStartSLOduration=2.74635619 podStartE2EDuration="2.74635619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:06.727992741 +0000 UTC m=+984.423348224" watchObservedRunningTime="2026-11-02 17:46:06.74635619 +0000 UTC m=+984.441711664" Nov 02 17:46:07.242339 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:07.242285 4471 patch_prober.go:29] interesting pod/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 17:46:07.242581 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:07.242557 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" podUID="1c7aaa66-3d4a-4462-82d4-faf4ed4345f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 17:46:07.415932 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:07.415646 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 17:46:07.417194 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:07.417157 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-7l8nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 17:46:07.417392 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:07.417371 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" podUID="1aa74f62-fe2f-4238-9f58-4a0a20f7892a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 17:46:07.511396 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:07.509519 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-223215e4-868c4668c5-8nh97"] Nov 02 17:46:07.515663 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:07.515631 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-223215e4-b78cf56ff-kj7gq"] Nov 02 17:46:08.429329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:08.424736 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-7l8nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 17:46:08.429329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:08.424812 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" podUID="1aa74f62-fe2f-4238-9f58-4a0a20f7892a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 17:46:08.654666 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:08.654615 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:46:08.659330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:08.656213 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:46:08.663322 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:08.662549 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:46:09.353196 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:46:09.353156 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:46:09.415220 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:09.415176 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-7l8nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 17:46:09.415413 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:09.415244 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" podUID="1aa74f62-fe2f-4238-9f58-4a0a20f7892a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 17:46:10.213712 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:10.213676 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-556cd8a1-cb497c9c-bdlhx"] Nov 02 17:46:10.248285 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:10.248243 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-556cd8a1-5b4cc7856c-k289v"] Nov 02 17:46:10.416351 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:10.415315 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-7l8nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 17:46:10.416351 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:10.415384 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" podUID="1aa74f62-fe2f-4238-9f58-4a0a20f7892a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 17:46:11.417329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:11.416714 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-7l8nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 17:46:11.417329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:11.416775 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" podUID="1aa74f62-fe2f-4238-9f58-4a0a20f7892a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 17:46:12.414869 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:12.414832 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-7l8nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 17:46:12.415080 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:12.414909 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" podUID="1aa74f62-fe2f-4238-9f58-4a0a20f7892a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 17:46:12.510204 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:12.510127 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-223215e4-868c4668c5-8nh97" podUID="10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2" containerName="payload-processing" containerID="cri-o://81f9001cb3fe13e7bdcba18483c03d0fb5f09c7ff04bbf8a3dab86b082b3de69" gracePeriod=25 Nov 02 17:46:12.516381 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:12.516017 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-223215e4-b78cf56ff-kj7gq" podUID="809eb3bd-2daf-40fe-a332-43032c2be57a" containerName="payload-pre-processing" containerID="cri-o://b8b628787f008202a574cda0da351bafbc0b7fdc85a42479566531f3e2dc0965" gracePeriod=25 Nov 02 17:46:12.691624 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:12.691529 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-223215e4-59469596d7-2lmwt"] Nov 02 17:46:12.691833 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:12.691805 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-223215e4-59469596d7-2lmwt" podUID="8278f2e9-f3b2-4609-a4a2-6db72c7647a4" containerName="maas-api" containerID="cri-o://bca336d5bd0b204854cc5b7b37bc2771ea795bf397109cf3ccbb7bc10a952f14" gracePeriod=30 Nov 02 17:46:12.891979 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:12.891525 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-223215e4-868c4668c5-8nh97" Nov 02 17:46:12.895641 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:12.895616 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-223215e4-b78cf56ff-kj7gq" Nov 02 17:46:12.967114 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:12.965792 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/809eb3bd-2daf-40fe-a332-43032c2be57a-kube-api-access-w56xx\" (UniqueName: \"kubernetes.io/projected/809eb3bd-2daf-40fe-a332-43032c2be57a-kube-api-access-w56xx\") pod \"809eb3bd-2daf-40fe-a332-43032c2be57a\" (UID: \"809eb3bd-2daf-40fe-a332-43032c2be57a\") " Nov 02 17:46:12.967114 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:12.965959 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2-kube-api-access-qbvlx\" (UniqueName: \"kubernetes.io/projected/10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2-kube-api-access-qbvlx\") pod \"10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2\" (UID: \"10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2\") " Nov 02 17:46:12.967114 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:12.966120 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2-plugins-config-volume\") pod \"10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2\" (UID: \"10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2\") " Nov 02 17:46:12.967114 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:12.966155 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/809eb3bd-2daf-40fe-a332-43032c2be57a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/809eb3bd-2daf-40fe-a332-43032c2be57a-plugins-config-volume\") pod \"809eb3bd-2daf-40fe-a332-43032c2be57a\" (UID: \"809eb3bd-2daf-40fe-a332-43032c2be57a\") " Nov 02 17:46:13.026469 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.015562 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/809eb3bd-2daf-40fe-a332-43032c2be57a-kube-api-access-w56xx" pod "809eb3bd-2daf-40fe-a332-43032c2be57a" (UID: "809eb3bd-2daf-40fe-a332-43032c2be57a"). InnerVolumeSpecName "kube-api-access-w56xx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:46:13.058386 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.031787 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/809eb3bd-2daf-40fe-a332-43032c2be57a-plugins-config-volume" pod "809eb3bd-2daf-40fe-a332-43032c2be57a" (UID: "809eb3bd-2daf-40fe-a332-43032c2be57a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:46:13.092412 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.058817 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2-plugins-config-volume" pod "10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2" (UID: "10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:46:13.110256 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.100763 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w56xx\" (UniqueName: \"kubernetes.io/projected/809eb3bd-2daf-40fe-a332-43032c2be57a-kube-api-access-w56xx\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:46:13.110256 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.101399 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:46:13.110256 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.101427 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/809eb3bd-2daf-40fe-a332-43032c2be57a-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:46:13.110256 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.105405 4471 generic.go:356] "Generic (PLEG): container finished" podID="8278f2e9-f3b2-4609-a4a2-6db72c7647a4" containerID="bca336d5bd0b204854cc5b7b37bc2771ea795bf397109cf3ccbb7bc10a952f14" exitCode=0 Nov 02 17:46:13.110256 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.105637 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-223215e4-59469596d7-2lmwt" event={"ID":"8278f2e9-f3b2-4609-a4a2-6db72c7647a4","Type":"ContainerDied","Data":"bca336d5bd0b204854cc5b7b37bc2771ea795bf397109cf3ccbb7bc10a952f14"} Nov 02 17:46:13.110256 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.107741 4471 generic.go:356] "Generic (PLEG): container finished" podID="809eb3bd-2daf-40fe-a332-43032c2be57a" containerID="b8b628787f008202a574cda0da351bafbc0b7fdc85a42479566531f3e2dc0965" exitCode=0 Nov 02 17:46:13.110256 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.107831 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-223215e4-b78cf56ff-kj7gq" event={"ID":"809eb3bd-2daf-40fe-a332-43032c2be57a","Type":"ContainerDied","Data":"b8b628787f008202a574cda0da351bafbc0b7fdc85a42479566531f3e2dc0965"} Nov 02 17:46:13.110256 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.107877 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-223215e4-b78cf56ff-kj7gq" event={"ID":"809eb3bd-2daf-40fe-a332-43032c2be57a","Type":"ContainerDied","Data":"5219fc63202660ef28f2feed609bebc9536f137258e7faeb6eb463e8bb91c454"} Nov 02 17:46:13.110256 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.107908 4471 scope.go:122] "RemoveContainer" containerID="b8b628787f008202a574cda0da351bafbc0b7fdc85a42479566531f3e2dc0965" Nov 02 17:46:13.110256 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.108163 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-223215e4-b78cf56ff-kj7gq" Nov 02 17:46:13.116701 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.116675 4471 generic.go:356] "Generic (PLEG): container finished" podID="10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2" containerID="81f9001cb3fe13e7bdcba18483c03d0fb5f09c7ff04bbf8a3dab86b082b3de69" exitCode=0 Nov 02 17:46:13.117352 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.117334 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-223215e4-868c4668c5-8nh97" Nov 02 17:46:13.117658 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.117638 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-223215e4-868c4668c5-8nh97" event={"ID":"10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2","Type":"ContainerDied","Data":"81f9001cb3fe13e7bdcba18483c03d0fb5f09c7ff04bbf8a3dab86b082b3de69"} Nov 02 17:46:13.117818 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.117801 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-223215e4-868c4668c5-8nh97" event={"ID":"10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2","Type":"ContainerDied","Data":"18e0edd1d42081d947f581ab3923cff6db89d3aba00dc8970153080e9ee716d3"} Nov 02 17:46:13.122436 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.122411 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2-kube-api-access-qbvlx" pod "10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2" (UID: "10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2"). InnerVolumeSpecName "kube-api-access-qbvlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:46:13.207394 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.207355 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbvlx\" (UniqueName: \"kubernetes.io/projected/10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2-kube-api-access-qbvlx\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:46:13.269574 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.260275 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-223215e4-b78cf56ff-kj7gq"] Nov 02 17:46:13.269574 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.265780 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-223215e4-b78cf56ff-kj7gq"] Nov 02 17:46:13.444517 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.444484 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-7l8nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 17:46:13.456179 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.455889 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" podUID="1aa74f62-fe2f-4238-9f58-4a0a20f7892a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 17:46:13.557218 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.557192 4471 scope.go:122] "RemoveContainer" containerID="b8b628787f008202a574cda0da351bafbc0b7fdc85a42479566531f3e2dc0965" Nov 02 17:46:13.569589 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:46:13.566140 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8b628787f008202a574cda0da351bafbc0b7fdc85a42479566531f3e2dc0965\": container with ID starting with b8b628787f008202a574cda0da351bafbc0b7fdc85a42479566531f3e2dc0965 not found: ID does not exist" containerID="b8b628787f008202a574cda0da351bafbc0b7fdc85a42479566531f3e2dc0965" Nov 02 17:46:13.569589 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.566186 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8b628787f008202a574cda0da351bafbc0b7fdc85a42479566531f3e2dc0965"} err="failed to get container status \"b8b628787f008202a574cda0da351bafbc0b7fdc85a42479566531f3e2dc0965\": rpc error: code = NotFound desc = could not find container \"b8b628787f008202a574cda0da351bafbc0b7fdc85a42479566531f3e2dc0965\": container with ID starting with b8b628787f008202a574cda0da351bafbc0b7fdc85a42479566531f3e2dc0965 not found: ID does not exist" Nov 02 17:46:13.569589 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.566211 4471 scope.go:122] "RemoveContainer" containerID="81f9001cb3fe13e7bdcba18483c03d0fb5f09c7ff04bbf8a3dab86b082b3de69" Nov 02 17:46:13.662300 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.662264 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-223215e4-868c4668c5-8nh97"] Nov 02 17:46:13.756564 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.756530 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-223215e4-59469596d7-2lmwt" Nov 02 17:46:13.786984 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.786884 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-223215e4-868c4668c5-8nh97"] Nov 02 17:46:13.821095 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.821062 4471 scope.go:122] "RemoveContainer" containerID="81f9001cb3fe13e7bdcba18483c03d0fb5f09c7ff04bbf8a3dab86b082b3de69" Nov 02 17:46:13.821936 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:46:13.821905 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81f9001cb3fe13e7bdcba18483c03d0fb5f09c7ff04bbf8a3dab86b082b3de69\": container with ID starting with 81f9001cb3fe13e7bdcba18483c03d0fb5f09c7ff04bbf8a3dab86b082b3de69 not found: ID does not exist" containerID="81f9001cb3fe13e7bdcba18483c03d0fb5f09c7ff04bbf8a3dab86b082b3de69" Nov 02 17:46:13.833999 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.833954 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81f9001cb3fe13e7bdcba18483c03d0fb5f09c7ff04bbf8a3dab86b082b3de69"} err="failed to get container status \"81f9001cb3fe13e7bdcba18483c03d0fb5f09c7ff04bbf8a3dab86b082b3de69\": rpc error: code = NotFound desc = could not find container \"81f9001cb3fe13e7bdcba18483c03d0fb5f09c7ff04bbf8a3dab86b082b3de69\": container with ID starting with 81f9001cb3fe13e7bdcba18483c03d0fb5f09c7ff04bbf8a3dab86b082b3de69 not found: ID does not exist" Nov 02 17:46:13.834218 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.833528 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8278f2e9-f3b2-4609-a4a2-6db72c7647a4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8278f2e9-f3b2-4609-a4a2-6db72c7647a4-maas-api-tls\") pod \"8278f2e9-f3b2-4609-a4a2-6db72c7647a4\" (UID: \"8278f2e9-f3b2-4609-a4a2-6db72c7647a4\") " Nov 02 17:46:13.834487 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.834463 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8278f2e9-f3b2-4609-a4a2-6db72c7647a4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8278f2e9-f3b2-4609-a4a2-6db72c7647a4-metrics-certs\") pod \"8278f2e9-f3b2-4609-a4a2-6db72c7647a4\" (UID: \"8278f2e9-f3b2-4609-a4a2-6db72c7647a4\") " Nov 02 17:46:13.834733 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.834714 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8278f2e9-f3b2-4609-a4a2-6db72c7647a4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8278f2e9-f3b2-4609-a4a2-6db72c7647a4-openshift-service-ca\") pod \"8278f2e9-f3b2-4609-a4a2-6db72c7647a4\" (UID: \"8278f2e9-f3b2-4609-a4a2-6db72c7647a4\") " Nov 02 17:46:13.834966 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.834945 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8278f2e9-f3b2-4609-a4a2-6db72c7647a4-kube-api-access-8wzhs\" (UniqueName: \"kubernetes.io/projected/8278f2e9-f3b2-4609-a4a2-6db72c7647a4-kube-api-access-8wzhs\") pod \"8278f2e9-f3b2-4609-a4a2-6db72c7647a4\" (UID: \"8278f2e9-f3b2-4609-a4a2-6db72c7647a4\") " Nov 02 17:46:13.862341 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.860471 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8278f2e9-f3b2-4609-a4a2-6db72c7647a4-maas-api-tls" pod "8278f2e9-f3b2-4609-a4a2-6db72c7647a4" (UID: "8278f2e9-f3b2-4609-a4a2-6db72c7647a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:46:13.891955 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.868850 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8278f2e9-f3b2-4609-a4a2-6db72c7647a4-openshift-service-ca" pod "8278f2e9-f3b2-4609-a4a2-6db72c7647a4" (UID: "8278f2e9-f3b2-4609-a4a2-6db72c7647a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:46:13.956362 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.940999 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8278f2e9-f3b2-4609-a4a2-6db72c7647a4-maas-api-tls\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:46:13.956362 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:13.941040 4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8278f2e9-f3b2-4609-a4a2-6db72c7647a4-openshift-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:46:14.064858 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:14.020093 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8278f2e9-f3b2-4609-a4a2-6db72c7647a4-metrics-certs" pod "8278f2e9-f3b2-4609-a4a2-6db72c7647a4" (UID: "8278f2e9-f3b2-4609-a4a2-6db72c7647a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:46:14.078710 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:14.078668 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8278f2e9-f3b2-4609-a4a2-6db72c7647a4-kube-api-access-8wzhs" pod "8278f2e9-f3b2-4609-a4a2-6db72c7647a4" (UID: "8278f2e9-f3b2-4609-a4a2-6db72c7647a4"). InnerVolumeSpecName "kube-api-access-8wzhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:46:14.150330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:14.114985 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wzhs\" (UniqueName: \"kubernetes.io/projected/8278f2e9-f3b2-4609-a4a2-6db72c7647a4-kube-api-access-8wzhs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:46:14.150330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:14.137687 4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8278f2e9-f3b2-4609-a4a2-6db72c7647a4-metrics-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:46:14.292049 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:14.288429 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-223215e4-59469596d7-2lmwt" Nov 02 17:46:14.292049 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:14.288742 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-223215e4-59469596d7-2lmwt" event={"ID":"8278f2e9-f3b2-4609-a4a2-6db72c7647a4","Type":"ContainerDied","Data":"f082b2de3450cae68738c3f888fce0e0ffc3c45323735078d4276b12a05b2b22"} Nov 02 17:46:14.292049 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:14.288810 4471 scope.go:122] "RemoveContainer" containerID="bca336d5bd0b204854cc5b7b37bc2771ea795bf397109cf3ccbb7bc10a952f14" Nov 02 17:46:14.363654 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:14.363619 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-223215e4-59469596d7-2lmwt"] Nov 02 17:46:14.379808 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:14.379763 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-223215e4-59469596d7-2lmwt"] Nov 02 17:46:14.415871 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:14.415834 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-7l8nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 17:46:14.416020 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:14.415901 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" podUID="1aa74f62-fe2f-4238-9f58-4a0a20f7892a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 17:46:14.446227 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:14.446183 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2" path="/var/lib/kubelet/pods/10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2/volumes" Nov 02 17:46:14.446849 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:14.446834 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="809eb3bd-2daf-40fe-a332-43032c2be57a" path="/var/lib/kubelet/pods/809eb3bd-2daf-40fe-a332-43032c2be57a/volumes" Nov 02 17:46:14.447288 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:14.447268 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8278f2e9-f3b2-4609-a4a2-6db72c7647a4" path="/var/lib/kubelet/pods/8278f2e9-f3b2-4609-a4a2-6db72c7647a4/volumes" Nov 02 17:46:15.214092 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.214022 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-556cd8a1-cb497c9c-bdlhx" podUID="15f5643c-8167-4504-ba33-51b35aa781d7" containerName="payload-processing" containerID="cri-o://cee3288066dbc33c1586e32b762cb1b55d82a33839852109183a4a56083f8d2d" gracePeriod=25 Nov 02 17:46:15.248988 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.248945 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-556cd8a1-5b4cc7856c-k289v" podUID="7ad02b4a-be6d-4f85-b8fc-8e9188e35649" containerName="payload-pre-processing" containerID="cri-o://7138748a51e3bb067098d3a466cba592fe020a1ef9adb661d92451dec4c85def" gracePeriod=25 Nov 02 17:46:15.330645 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.324410 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt"] Nov 02 17:46:15.330645 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.325534 4471 cpu_manager.go:414] "containerMap: removing container" podUID="8278f2e9-f3b2-4609-a4a2-6db72c7647a4" containerName="maas-api" Nov 02 17:46:15.330645 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.325816 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8278f2e9-f3b2-4609-a4a2-6db72c7647a4" containerName="maas-api" Nov 02 17:46:15.330645 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.325875 4471 cpu_manager.go:414] "containerMap: removing container" podUID="809eb3bd-2daf-40fe-a332-43032c2be57a" containerName="payload-pre-processing" Nov 02 17:46:15.330645 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.325885 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="809eb3bd-2daf-40fe-a332-43032c2be57a" containerName="payload-pre-processing" Nov 02 17:46:15.330645 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.325935 4471 cpu_manager.go:414] "containerMap: removing container" podUID="10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2" containerName="payload-processing" Nov 02 17:46:15.330645 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.325944 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2" containerName="payload-processing" Nov 02 17:46:15.330645 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.326113 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="809eb3bd-2daf-40fe-a332-43032c2be57a" containerName="payload-pre-processing" Nov 02 17:46:15.330645 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.326134 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="10d8c0f7-b825-4d6d-9fd4-e60be9d17ff2" containerName="payload-processing" Nov 02 17:46:15.330645 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.326152 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="8278f2e9-f3b2-4609-a4a2-6db72c7647a4" containerName="maas-api" Nov 02 17:46:15.336467 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.333591 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:46:15.339227 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.339187 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt"] Nov 02 17:46:15.366209 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.366169 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-ea104436-gw-openshift-default-dockercfg-kcbt8\"" Nov 02 17:46:15.406787 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.406752 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-credential-socket\") pod \"e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:46:15.406943 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.406916 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-istio-data\") pod \"e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:46:15.406943 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.406939 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-workload-certs\") pod \"e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:46:15.407082 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.406998 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-istio-envoy\") pod \"e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:46:15.407082 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.407065 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/613f1277-5dc5-422e-8b92-d9be8ac858c4-istiod-ca-cert\") pod \"e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:46:15.407191 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.407104 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/613f1277-5dc5-422e-8b92-d9be8ac858c4-istio-token\") pod \"e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:46:15.407191 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.407139 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-workload-socket\") pod \"e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:46:15.407191 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.407187 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/613f1277-5dc5-422e-8b92-d9be8ac858c4-istio-podinfo\") pod \"e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:46:15.407399 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.407221 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4jm7\" (UniqueName: \"kubernetes.io/projected/613f1277-5dc5-422e-8b92-d9be8ac858c4-kube-api-access-n4jm7\") pod \"e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:46:15.508972 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.508818 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-istio-data\") pod \"e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:46:15.508972 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.508876 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-workload-certs\") pod \"e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:46:15.509691 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.509646 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-istio-data\") pod \"e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:46:15.509837 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.509783 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-workload-certs\") pod \"e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:46:15.510131 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.510044 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-istio-envoy\") pod \"e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:46:15.510931 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.510815 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/613f1277-5dc5-422e-8b92-d9be8ac858c4-istiod-ca-cert\") pod \"e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:46:15.510931 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.510878 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/613f1277-5dc5-422e-8b92-d9be8ac858c4-istio-token\") pod \"e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:46:15.511530 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.511166 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-workload-socket\") pod \"e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:46:15.511530 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.511244 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/613f1277-5dc5-422e-8b92-d9be8ac858c4-istio-podinfo\") pod \"e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:46:15.511530 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.511293 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4jm7\" (UniqueName: \"kubernetes.io/projected/613f1277-5dc5-422e-8b92-d9be8ac858c4-kube-api-access-n4jm7\") pod \"e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:46:15.511530 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.511374 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-credential-socket\") pod \"e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:46:15.512998 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.512374 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-workload-socket\") pod \"e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:46:15.512998 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.512597 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-credential-socket\") pod \"e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:46:15.513428 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.513385 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/613f1277-5dc5-422e-8b92-d9be8ac858c4-istiod-ca-cert\") pod \"e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:46:15.530329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.527795 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-istio-envoy\") pod \"e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:46:15.530329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.530181 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/613f1277-5dc5-422e-8b92-d9be8ac858c4-istio-podinfo\") pod \"e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:46:15.543117 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.542947 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/613f1277-5dc5-422e-8b92-d9be8ac858c4-istio-token\") pod \"e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:46:15.543117 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.542994 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4jm7\" (UniqueName: \"kubernetes.io/projected/613f1277-5dc5-422e-8b92-d9be8ac858c4-kube-api-access-n4jm7\") pod \"e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:46:15.676917 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.676588 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-556cd8a1-cb497c9c-bdlhx" Nov 02 17:46:15.682653 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.682622 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:46:15.935084 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.909416 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15f5643c-8167-4504-ba33-51b35aa781d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f5643c-8167-4504-ba33-51b35aa781d7-plugins-config-volume\") pod \"15f5643c-8167-4504-ba33-51b35aa781d7\" (UID: \"15f5643c-8167-4504-ba33-51b35aa781d7\") " Nov 02 17:46:15.935084 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.909501 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15f5643c-8167-4504-ba33-51b35aa781d7-kube-api-access-f4jzl\" (UniqueName: \"kubernetes.io/projected/15f5643c-8167-4504-ba33-51b35aa781d7-kube-api-access-f4jzl\") pod \"15f5643c-8167-4504-ba33-51b35aa781d7\" (UID: \"15f5643c-8167-4504-ba33-51b35aa781d7\") " Nov 02 17:46:15.935084 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.921386 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15f5643c-8167-4504-ba33-51b35aa781d7-plugins-config-volume" pod "15f5643c-8167-4504-ba33-51b35aa781d7" (UID: "15f5643c-8167-4504-ba33-51b35aa781d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:46:15.950526 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.950468 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f5643c-8167-4504-ba33-51b35aa781d7-kube-api-access-f4jzl" pod "15f5643c-8167-4504-ba33-51b35aa781d7" (UID: "15f5643c-8167-4504-ba33-51b35aa781d7"). InnerVolumeSpecName "kube-api-access-f4jzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:46:15.999359 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:15.988898 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-556cd8a1-5b4cc7856c-k289v" Nov 02 17:46:16.059333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:16.046703 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f5643c-8167-4504-ba33-51b35aa781d7-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:46:16.059333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:16.046738 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4jzl\" (UniqueName: \"kubernetes.io/projected/15f5643c-8167-4504-ba33-51b35aa781d7-kube-api-access-f4jzl\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:46:16.084928 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:16.084424 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-223215e4-59469596d7-2lmwt" podUID="8278f2e9-f3b2-4609-a4a2-6db72c7647a4" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.82:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:16.206550 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:16.170410 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ad02b4a-be6d-4f85-b8fc-8e9188e35649-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ad02b4a-be6d-4f85-b8fc-8e9188e35649-plugins-config-volume\") pod \"7ad02b4a-be6d-4f85-b8fc-8e9188e35649\" (UID: \"7ad02b4a-be6d-4f85-b8fc-8e9188e35649\") " Nov 02 17:46:16.206550 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:16.170547 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ad02b4a-be6d-4f85-b8fc-8e9188e35649-kube-api-access-vt4xz\" (UniqueName: \"kubernetes.io/projected/7ad02b4a-be6d-4f85-b8fc-8e9188e35649-kube-api-access-vt4xz\") pod \"7ad02b4a-be6d-4f85-b8fc-8e9188e35649\" (UID: \"7ad02b4a-be6d-4f85-b8fc-8e9188e35649\") " Nov 02 17:46:16.206550 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:16.203082 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ad02b4a-be6d-4f85-b8fc-8e9188e35649-plugins-config-volume" pod "7ad02b4a-be6d-4f85-b8fc-8e9188e35649" (UID: "7ad02b4a-be6d-4f85-b8fc-8e9188e35649"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:46:16.219515 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:16.218369 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad02b4a-be6d-4f85-b8fc-8e9188e35649-kube-api-access-vt4xz" pod "7ad02b4a-be6d-4f85-b8fc-8e9188e35649" (UID: "7ad02b4a-be6d-4f85-b8fc-8e9188e35649"). InnerVolumeSpecName "kube-api-access-vt4xz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:46:16.300758 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:16.300690 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ad02b4a-be6d-4f85-b8fc-8e9188e35649-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:46:16.300758 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:16.300727 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt4xz\" (UniqueName: \"kubernetes.io/projected/7ad02b4a-be6d-4f85-b8fc-8e9188e35649-kube-api-access-vt4xz\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:46:16.627938 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:16.627895 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-7l8nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:46:16.628227 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:16.628191 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" podUID="1aa74f62-fe2f-4238-9f58-4a0a20f7892a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:16.821020 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:16.820984 4471 generic.go:356] "Generic (PLEG): container finished" podID="7ad02b4a-be6d-4f85-b8fc-8e9188e35649" containerID="7138748a51e3bb067098d3a466cba592fe020a1ef9adb661d92451dec4c85def" exitCode=0 Nov 02 17:46:16.821371 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:16.821347 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-556cd8a1-5b4cc7856c-k289v" Nov 02 17:46:16.823333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:16.823089 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-556cd8a1-5b4cc7856c-k289v" event={"ID":"7ad02b4a-be6d-4f85-b8fc-8e9188e35649","Type":"ContainerDied","Data":"7138748a51e3bb067098d3a466cba592fe020a1ef9adb661d92451dec4c85def"} Nov 02 17:46:16.823333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:16.823154 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt"] Nov 02 17:46:16.823333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:16.823190 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-556cd8a1-5b4cc7856c-k289v" event={"ID":"7ad02b4a-be6d-4f85-b8fc-8e9188e35649","Type":"ContainerDied","Data":"643862ed48eb5c34b0106a9bb4ac4bbc3e4ffa71616f8efbb04953d359945bdb"} Nov 02 17:46:16.823333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:16.823224 4471 scope.go:122] "RemoveContainer" containerID="7138748a51e3bb067098d3a466cba592fe020a1ef9adb661d92451dec4c85def" Nov 02 17:46:16.835067 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:16.835037 4471 generic.go:356] "Generic (PLEG): container finished" podID="15f5643c-8167-4504-ba33-51b35aa781d7" containerID="cee3288066dbc33c1586e32b762cb1b55d82a33839852109183a4a56083f8d2d" exitCode=0 Nov 02 17:46:16.835524 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:16.835247 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-556cd8a1-cb497c9c-bdlhx" event={"ID":"15f5643c-8167-4504-ba33-51b35aa781d7","Type":"ContainerDied","Data":"cee3288066dbc33c1586e32b762cb1b55d82a33839852109183a4a56083f8d2d"} Nov 02 17:46:16.835524 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:16.835338 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-556cd8a1-cb497c9c-bdlhx" event={"ID":"15f5643c-8167-4504-ba33-51b35aa781d7","Type":"ContainerDied","Data":"7670dbcbf41b890702554ceee55ee90255a3ec61d5dcf5f4ff551621214fba48"} Nov 02 17:46:16.835524 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:16.835425 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-556cd8a1-cb497c9c-bdlhx" Nov 02 17:46:16.836998 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:16.836960 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:46:16.837108 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:16.837049 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:46:16.837108 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:16.837094 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:46:16.847044 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:16.847023 4471 scope.go:122] "RemoveContainer" containerID="7138748a51e3bb067098d3a466cba592fe020a1ef9adb661d92451dec4c85def" Nov 02 17:46:16.847397 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:46:16.847371 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7138748a51e3bb067098d3a466cba592fe020a1ef9adb661d92451dec4c85def\": container with ID starting with 7138748a51e3bb067098d3a466cba592fe020a1ef9adb661d92451dec4c85def not found: ID does not exist" containerID="7138748a51e3bb067098d3a466cba592fe020a1ef9adb661d92451dec4c85def" Nov 02 17:46:16.847504 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:16.847401 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7138748a51e3bb067098d3a466cba592fe020a1ef9adb661d92451dec4c85def"} err="failed to get container status \"7138748a51e3bb067098d3a466cba592fe020a1ef9adb661d92451dec4c85def\": rpc error: code = NotFound desc = could not find container \"7138748a51e3bb067098d3a466cba592fe020a1ef9adb661d92451dec4c85def\": container with ID starting with 7138748a51e3bb067098d3a466cba592fe020a1ef9adb661d92451dec4c85def not found: ID does not exist" Nov 02 17:46:16.847504 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:16.847421 4471 scope.go:122] "RemoveContainer" containerID="cee3288066dbc33c1586e32b762cb1b55d82a33839852109183a4a56083f8d2d" Nov 02 17:46:16.858562 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:16.858539 4471 scope.go:122] "RemoveContainer" containerID="cee3288066dbc33c1586e32b762cb1b55d82a33839852109183a4a56083f8d2d" Nov 02 17:46:16.858885 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:46:16.858861 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cee3288066dbc33c1586e32b762cb1b55d82a33839852109183a4a56083f8d2d\": container with ID starting with cee3288066dbc33c1586e32b762cb1b55d82a33839852109183a4a56083f8d2d not found: ID does not exist" containerID="cee3288066dbc33c1586e32b762cb1b55d82a33839852109183a4a56083f8d2d" Nov 02 17:46:16.858959 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:16.858897 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cee3288066dbc33c1586e32b762cb1b55d82a33839852109183a4a56083f8d2d"} err="failed to get container status \"cee3288066dbc33c1586e32b762cb1b55d82a33839852109183a4a56083f8d2d\": rpc error: code = NotFound desc = could not find container \"cee3288066dbc33c1586e32b762cb1b55d82a33839852109183a4a56083f8d2d\": container with ID starting with cee3288066dbc33c1586e32b762cb1b55d82a33839852109183a4a56083f8d2d not found: ID does not exist" Nov 02 17:46:16.904809 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:16.904782 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-556cd8a1-5b4cc7856c-k289v"] Nov 02 17:46:16.910167 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:16.910144 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-556cd8a1-5b4cc7856c-k289v"] Nov 02 17:46:16.952418 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:16.952376 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-556cd8a1-cb497c9c-bdlhx"] Nov 02 17:46:16.954025 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:16.953997 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-556cd8a1-cb497c9c-bdlhx"] Nov 02 17:46:17.006062 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:17.006031 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 17:46:17.007103 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:17.007083 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 17:46:17.007873 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:17.007854 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 17:46:17.848857 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:17.848813 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" event={"ID":"613f1277-5dc5-422e-8b92-d9be8ac858c4","Type":"ContainerStarted","Data":"053c1cc3d43c4bb595a4e790118314ee2260d473843fe0590ff297f6845112d7"} Nov 02 17:46:17.849380 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:17.848928 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" event={"ID":"613f1277-5dc5-422e-8b92-d9be8ac858c4","Type":"ContainerStarted","Data":"9af1aa617392a9321de03772c236a80ccedc1a8052ca420a99e4e6f25544c89a"} Nov 02 17:46:17.857032 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:17.856999 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx"] Nov 02 17:46:17.857401 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:17.857374 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" podUID="1c7aaa66-3d4a-4462-82d4-faf4ed4345f9" containerName="istio-proxy" containerID="cri-o://49c69119eb64eb39944e116cfaadca086eacb6203c20f4bb605867180932ce2b" gracePeriod=30 Nov 02 17:46:17.901570 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:17.901521 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" podStartSLOduration=2.9015071260000003 podStartE2EDuration="2.901507126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:17.8990428 +0000 UTC m=+995.594398272" watchObservedRunningTime="2026-11-02 17:46:17.901507126 +0000 UTC m=+995.596862675" Nov 02 17:46:18.504337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:18.496036 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15f5643c-8167-4504-ba33-51b35aa781d7" path="/var/lib/kubelet/pods/15f5643c-8167-4504-ba33-51b35aa781d7/volumes" Nov 02 17:46:18.504337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:18.496834 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ad02b4a-be6d-4f85-b8fc-8e9188e35649" path="/var/lib/kubelet/pods/7ad02b4a-be6d-4f85-b8fc-8e9188e35649/volumes" Nov 02 17:46:18.683261 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:18.683224 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:46:18.689471 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:18.689444 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:46:18.857703 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:18.857664 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:46:18.858457 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:18.858439 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:46:19.059675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.059640 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx"] Nov 02 17:46:19.060759 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.060731 4471 cpu_manager.go:414] "containerMap: removing container" podUID="15f5643c-8167-4504-ba33-51b35aa781d7" containerName="payload-processing" Nov 02 17:46:19.060759 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.060758 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15f5643c-8167-4504-ba33-51b35aa781d7" containerName="payload-processing" Nov 02 17:46:19.060955 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.060780 4471 cpu_manager.go:414] "containerMap: removing container" podUID="7ad02b4a-be6d-4f85-b8fc-8e9188e35649" containerName="payload-pre-processing" Nov 02 17:46:19.060955 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.060788 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ad02b4a-be6d-4f85-b8fc-8e9188e35649" containerName="payload-pre-processing" Nov 02 17:46:19.061061 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.060966 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="15f5643c-8167-4504-ba33-51b35aa781d7" containerName="payload-processing" Nov 02 17:46:19.061061 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.060980 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ad02b4a-be6d-4f85-b8fc-8e9188e35649" containerName="payload-pre-processing" Nov 02 17:46:19.064824 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.064801 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:46:19.090587 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.090560 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-93321a4f-openshift-default-dockercfg-8fp78\"" Nov 02 17:46:19.091219 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.091196 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx"] Nov 02 17:46:19.140127 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.140047 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istio-data\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:46:19.140127 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.140087 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-workload-socket\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:46:19.140400 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.140232 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-workload-certs\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:46:19.140400 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.140346 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istio-token\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:46:19.140498 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.140421 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-credential-socket\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:46:19.140498 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.140481 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istio-podinfo\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:46:19.140651 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.140565 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istio-envoy\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:46:19.140651 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.140599 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f69c6\" (UniqueName: \"kubernetes.io/projected/7ebdad0f-3d23-452f-83dc-55fed8615cf7-kube-api-access-f69c6\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:46:19.140746 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.140684 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istiod-ca-cert\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:46:19.245325 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.244355 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istio-podinfo\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:46:19.245325 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.244440 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istio-envoy\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:46:19.245325 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.244481 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f69c6\" (UniqueName: \"kubernetes.io/projected/7ebdad0f-3d23-452f-83dc-55fed8615cf7-kube-api-access-f69c6\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:46:19.245325 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.244550 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istiod-ca-cert\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:46:19.245325 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.244629 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istio-data\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:46:19.245325 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.244665 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-workload-socket\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:46:19.245325 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.244785 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-workload-certs\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:46:19.245325 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.244856 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istio-token\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:46:19.245325 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.244949 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-credential-socket\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:46:19.267330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.265248 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istiod-ca-cert\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:46:19.267330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.265901 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-workload-socket\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:46:19.267330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.266220 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istio-data\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:46:19.289439 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.286383 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-workload-certs\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:46:19.289439 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.286591 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-credential-socket\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:46:19.323326 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.312581 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istio-envoy\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:46:19.323326 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.314808 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istio-podinfo\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:46:19.350380 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.343851 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istio-token\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:46:19.350380 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.344835 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f69c6\" (UniqueName: \"kubernetes.io/projected/7ebdad0f-3d23-452f-83dc-55fed8615cf7-kube-api-access-f69c6\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:46:19.381845 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.381803 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:46:19.989253 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.989173 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk"] Nov 02 17:46:19.993623 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.993601 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 17:46:19.996201 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:46:19.996176 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ebdad0f_3d23_452f_83dc_55fed8615cf7.slice/crio-edbb12364b3f0790b115b7f02ee2e23f6bae4feae74585caa8328d111aada85f WatchSource:0}: Error finding container edbb12364b3f0790b115b7f02ee2e23f6bae4feae74585caa8328d111aada85f: Status 404 returned error can't find the container with id edbb12364b3f0790b115b7f02ee2e23f6bae4feae74585caa8328d111aada85f Nov 02 17:46:19.999160 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.999128 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:46:19.999255 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.999201 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:46:19.999255 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:19.999245 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:46:20.001722 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.001700 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx"] Nov 02 17:46:20.019941 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.019911 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk"] Nov 02 17:46:20.038140 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.037715 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46"] Nov 02 17:46:20.044099 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.044073 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 17:46:20.069230 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.069195 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46"] Nov 02 17:46:20.111717 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.111683 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-gg9fk\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 17:46:20.111906 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.111732 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-k8n46\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 17:46:20.111906 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.111781 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-gg9fk\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 17:46:20.111906 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.111813 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-k8n46\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 17:46:20.111906 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.111834 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-gg9fk\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 17:46:20.111906 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.111859 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-k8n46\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 17:46:20.112175 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.111960 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd908c8a-4026-4da6-95c5-34112ecc019a-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-k8n46\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 17:46:20.112175 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.112009 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-gg9fk\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 17:46:20.112175 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.112044 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-gg9fk\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 17:46:20.112175 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.112100 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv6xv\" (UniqueName: \"kubernetes.io/projected/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-kube-api-access-vv6xv\") pod \"maas-default-gateway-openshift-default-76f4876667-gg9fk\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 17:46:20.112175 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.112162 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd908c8a-4026-4da6-95c5-34112ecc019a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-k8n46\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 17:46:20.112477 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.112326 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd908c8a-4026-4da6-95c5-34112ecc019a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-k8n46\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 17:46:20.112477 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.112367 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-gg9fk\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 17:46:20.112477 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.112400 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-gg9fk\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 17:46:20.112477 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.112446 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-k8n46\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 17:46:20.112668 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.112510 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-gg9fk\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 17:46:20.112668 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.112580 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-k8n46\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 17:46:20.112668 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.112613 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4c6k\" (UniqueName: \"kubernetes.io/projected/cd908c8a-4026-4da6-95c5-34112ecc019a-kube-api-access-m4c6k\") pod \"maas-default-gateway-openshift-default-76f4876667-k8n46\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 17:46:20.213739 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.213707 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd908c8a-4026-4da6-95c5-34112ecc019a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-k8n46\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 17:46:20.213739 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.213751 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-gg9fk\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 17:46:20.214017 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.213778 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-gg9fk\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 17:46:20.214017 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.213805 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-k8n46\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 17:46:20.214017 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.213855 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-gg9fk\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 17:46:20.214017 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.213898 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-k8n46\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 17:46:20.214017 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.213922 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4c6k\" (UniqueName: \"kubernetes.io/projected/cd908c8a-4026-4da6-95c5-34112ecc019a-kube-api-access-m4c6k\") pod \"maas-default-gateway-openshift-default-76f4876667-k8n46\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 17:46:20.214017 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.213957 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-gg9fk\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 17:46:20.214017 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.213982 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-k8n46\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 17:46:20.214017 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.214015 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-gg9fk\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 17:46:20.214482 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.214037 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-k8n46\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 17:46:20.214482 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.214053 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-gg9fk\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 17:46:20.214482 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.214077 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-k8n46\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 17:46:20.214482 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.214156 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd908c8a-4026-4da6-95c5-34112ecc019a-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-k8n46\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 17:46:20.214482 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.214202 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-gg9fk\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 17:46:20.214482 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.214244 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-gg9fk\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 17:46:20.214482 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.214300 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv6xv\" (UniqueName: \"kubernetes.io/projected/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-kube-api-access-vv6xv\") pod \"maas-default-gateway-openshift-default-76f4876667-gg9fk\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 17:46:20.214482 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.214380 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd908c8a-4026-4da6-95c5-34112ecc019a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-k8n46\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 17:46:20.214866 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.214537 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-k8n46\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 17:46:20.214866 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.214725 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-k8n46\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 17:46:20.214968 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.214921 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-gg9fk\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 17:46:20.215023 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.215004 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-gg9fk\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 17:46:20.215274 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.215252 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-k8n46\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 17:46:20.215527 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.215506 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-gg9fk\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 17:46:20.215735 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.215719 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-k8n46\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 17:46:20.216489 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.216285 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-gg9fk\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 17:46:20.216489 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.216364 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd908c8a-4026-4da6-95c5-34112ecc019a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-k8n46\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 17:46:20.216842 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.216667 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-gg9fk\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 17:46:20.217357 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.217297 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-gg9fk\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 17:46:20.217460 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.217368 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-gg9fk\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 17:46:20.217633 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.217599 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd908c8a-4026-4da6-95c5-34112ecc019a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-k8n46\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 17:46:20.218854 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.218833 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-k8n46\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 17:46:20.243632 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.243561 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv6xv\" (UniqueName: \"kubernetes.io/projected/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-kube-api-access-vv6xv\") pod \"maas-default-gateway-openshift-default-76f4876667-gg9fk\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 17:46:20.243779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.243655 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd908c8a-4026-4da6-95c5-34112ecc019a-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-k8n46\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 17:46:20.244491 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.244465 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-gg9fk\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 17:46:20.245548 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.245527 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4c6k\" (UniqueName: \"kubernetes.io/projected/cd908c8a-4026-4da6-95c5-34112ecc019a-kube-api-access-m4c6k\") pod \"maas-default-gateway-openshift-default-76f4876667-k8n46\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 17:46:20.310426 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.310388 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 17:46:20.383985 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.383667 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 17:46:20.678364 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:46:20.677772 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd14d1a7f_1dc5_4a6d_893b_03b53bc1cce5.slice/crio-797b0b336abf3e9668cd62a660c5f82a3cd86a3689a224697714a5ee3ff31d37 WatchSource:0}: Error finding container 797b0b336abf3e9668cd62a660c5f82a3cd86a3689a224697714a5ee3ff31d37: Status 404 returned error can't find the container with id 797b0b336abf3e9668cd62a660c5f82a3cd86a3689a224697714a5ee3ff31d37 Nov 02 17:46:20.686524 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.683270 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk"] Nov 02 17:46:20.700662 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.697263 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:46:20.700662 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.697372 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:46:20.700662 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.697443 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:46:20.753644 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:46:20.753615 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd908c8a_4026_4da6_95c5_34112ecc019a.slice/crio-838b02edd9f2504a916416325dbcc1514e4ff5d27ccb07f437321716eadc4ec6 WatchSource:0}: Error finding container 838b02edd9f2504a916416325dbcc1514e4ff5d27ccb07f437321716eadc4ec6: Status 404 returned error can't find the container with id 838b02edd9f2504a916416325dbcc1514e4ff5d27ccb07f437321716eadc4ec6 Nov 02 17:46:20.754265 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.754240 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46"] Nov 02 17:46:20.756726 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.756691 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:46:20.756982 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.756756 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:46:20.756982 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.756786 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:46:20.889348 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.878201 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" event={"ID":"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5","Type":"ContainerStarted","Data":"032035c86e70ca74d40491c92948cb2032b9912590738e4442690c6427481490"} Nov 02 17:46:20.889348 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.878266 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" event={"ID":"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5","Type":"ContainerStarted","Data":"797b0b336abf3e9668cd62a660c5f82a3cd86a3689a224697714a5ee3ff31d37"} Nov 02 17:46:20.889348 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.882046 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" event={"ID":"cd908c8a-4026-4da6-95c5-34112ecc019a","Type":"ContainerStarted","Data":"dd1da4ad3c90b90be0ec428ffbd80ac1faffbdaca8fedfc44005142de19bc009"} Nov 02 17:46:20.889348 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.882095 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" event={"ID":"cd908c8a-4026-4da6-95c5-34112ecc019a","Type":"ContainerStarted","Data":"838b02edd9f2504a916416325dbcc1514e4ff5d27ccb07f437321716eadc4ec6"} Nov 02 17:46:20.889348 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.884612 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" event={"ID":"7ebdad0f-3d23-452f-83dc-55fed8615cf7","Type":"ContainerStarted","Data":"178dae968066919b5d15597b76945c747d76c3251422482689e97b560bb9db6d"} Nov 02 17:46:20.889348 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.884685 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" event={"ID":"7ebdad0f-3d23-452f-83dc-55fed8615cf7","Type":"ContainerStarted","Data":"edbb12364b3f0790b115b7f02ee2e23f6bae4feae74585caa8328d111aada85f"} Nov 02 17:46:20.946392 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.942454 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" podStartSLOduration=1.942434795 podStartE2EDuration="1.942434795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:20.941451788 +0000 UTC m=+998.636807285" watchObservedRunningTime="2026-11-02 17:46:20.942434795 +0000 UTC m=+998.637790268" Nov 02 17:46:20.999787 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:20.999710 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" podStartSLOduration=1.999691569 podStartE2EDuration="1.999691569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:20.996192853 +0000 UTC m=+998.691548326" watchObservedRunningTime="2026-11-02 17:46:20.999691569 +0000 UTC m=+998.695047041" Nov 02 17:46:21.053147 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:21.053082 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" podStartSLOduration=2.053062239 podStartE2EDuration="2.053062239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:21.049795075 +0000 UTC m=+998.745150571" watchObservedRunningTime="2026-11-02 17:46:21.053062239 +0000 UTC m=+998.748417714" Nov 02 17:46:21.067704 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:21.067664 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ea104436-796f66c9bc-fdrgl"] Nov 02 17:46:21.072293 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:21.072261 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea104436-796f66c9bc-fdrgl" Nov 02 17:46:21.085342 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:21.085295 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ea104436-796f66c9bc-fdrgl"] Nov 02 17:46:21.092844 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:21.089284 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-ea104436-dockercfg-s8vmq\"" Nov 02 17:46:21.092844 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:21.089449 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-ea104436\"" Nov 02 17:46:21.152867 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:21.152824 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0891f7a9-f1df-49d5-b085-e2c343a05de5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ea104436-796f66c9bc-fdrgl\" (UID: \"0891f7a9-f1df-49d5-b085-e2c343a05de5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea104436-796f66c9bc-fdrgl" Nov 02 17:46:21.153065 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:21.152929 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jljb\" (UniqueName: \"kubernetes.io/projected/0891f7a9-f1df-49d5-b085-e2c343a05de5-kube-api-access-8jljb\") pod \"payload-pre-processing-e2e-migrate-ea104436-796f66c9bc-fdrgl\" (UID: \"0891f7a9-f1df-49d5-b085-e2c343a05de5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea104436-796f66c9bc-fdrgl" Nov 02 17:46:21.225378 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:21.222212 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ea104436-67cd4cf869-sk2pc"] Nov 02 17:46:21.375224 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:21.375183 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0891f7a9-f1df-49d5-b085-e2c343a05de5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ea104436-796f66c9bc-fdrgl\" (UID: \"0891f7a9-f1df-49d5-b085-e2c343a05de5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea104436-796f66c9bc-fdrgl" Nov 02 17:46:21.378662 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:21.378627 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jljb\" (UniqueName: \"kubernetes.io/projected/0891f7a9-f1df-49d5-b085-e2c343a05de5-kube-api-access-8jljb\") pod \"payload-pre-processing-e2e-migrate-ea104436-796f66c9bc-fdrgl\" (UID: \"0891f7a9-f1df-49d5-b085-e2c343a05de5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea104436-796f66c9bc-fdrgl" Nov 02 17:46:21.488744 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:21.469532 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gg9fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 17:46:21.488744 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:21.469619 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" podUID="d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 17:46:21.523703 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:21.523665 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ea104436-67cd4cf869-sk2pc"] Nov 02 17:46:21.527670 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:21.527643 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:46:21.527897 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:21.527883 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 17:46:21.528105 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:21.528092 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:46:21.528223 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:21.528213 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 17:46:21.528467 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:21.528438 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-k8n46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 17:46:21.528625 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:21.528598 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" podUID="cd908c8a-4026-4da6-95c5-34112ecc019a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 17:46:21.528837 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:21.527803 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ea104436-67cd4cf869-sk2pc" Nov 02 17:46:21.547177 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:21.547145 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0891f7a9-f1df-49d5-b085-e2c343a05de5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ea104436-796f66c9bc-fdrgl\" (UID: \"0891f7a9-f1df-49d5-b085-e2c343a05de5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea104436-796f66c9bc-fdrgl" Nov 02 17:46:21.573232 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:21.567073 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jljb\" (UniqueName: \"kubernetes.io/projected/0891f7a9-f1df-49d5-b085-e2c343a05de5-kube-api-access-8jljb\") pod \"payload-pre-processing-e2e-migrate-ea104436-796f66c9bc-fdrgl\" (UID: \"0891f7a9-f1df-49d5-b085-e2c343a05de5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea104436-796f66c9bc-fdrgl" Nov 02 17:46:21.664332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:21.664269 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1b56137-0532-4dff-bfbe-9ae561d67562-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ea104436-67cd4cf869-sk2pc\" (UID: \"b1b56137-0532-4dff-bfbe-9ae561d67562\") " pod="openshift-ingress/payload-processing-e2e-migrate-ea104436-67cd4cf869-sk2pc" Nov 02 17:46:21.664796 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:21.664759 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9k8w\" (UniqueName: \"kubernetes.io/projected/b1b56137-0532-4dff-bfbe-9ae561d67562-kube-api-access-r9k8w\") pod \"payload-processing-e2e-migrate-ea104436-67cd4cf869-sk2pc\" (UID: \"b1b56137-0532-4dff-bfbe-9ae561d67562\") " pod="openshift-ingress/payload-processing-e2e-migrate-ea104436-67cd4cf869-sk2pc" Nov 02 17:46:21.702673 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:21.702639 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea104436-796f66c9bc-fdrgl" Nov 02 17:46:21.766896 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:21.766852 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1b56137-0532-4dff-bfbe-9ae561d67562-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ea104436-67cd4cf869-sk2pc\" (UID: \"b1b56137-0532-4dff-bfbe-9ae561d67562\") " pod="openshift-ingress/payload-processing-e2e-migrate-ea104436-67cd4cf869-sk2pc" Nov 02 17:46:21.767094 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:21.767074 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9k8w\" (UniqueName: \"kubernetes.io/projected/b1b56137-0532-4dff-bfbe-9ae561d67562-kube-api-access-r9k8w\") pod \"payload-processing-e2e-migrate-ea104436-67cd4cf869-sk2pc\" (UID: \"b1b56137-0532-4dff-bfbe-9ae561d67562\") " pod="openshift-ingress/payload-processing-e2e-migrate-ea104436-67cd4cf869-sk2pc" Nov 02 17:46:21.767562 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:21.767540 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1b56137-0532-4dff-bfbe-9ae561d67562-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ea104436-67cd4cf869-sk2pc\" (UID: \"b1b56137-0532-4dff-bfbe-9ae561d67562\") " pod="openshift-ingress/payload-processing-e2e-migrate-ea104436-67cd4cf869-sk2pc" Nov 02 17:46:21.793189 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:21.793160 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9k8w\" (UniqueName: \"kubernetes.io/projected/b1b56137-0532-4dff-bfbe-9ae561d67562-kube-api-access-r9k8w\") pod \"payload-processing-e2e-migrate-ea104436-67cd4cf869-sk2pc\" (UID: \"b1b56137-0532-4dff-bfbe-9ae561d67562\") " pod="openshift-ingress/payload-processing-e2e-migrate-ea104436-67cd4cf869-sk2pc" Nov 02 17:46:21.892197 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:21.892161 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:46:21.892892 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:21.892866 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:46:21.916569 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:21.916547 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ea104436-67cd4cf869-sk2pc" Nov 02 17:46:22.016166 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:46:22.014770 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0891f7a9_f1df_49d5_b085_e2c343a05de5.slice/crio-e9d4ac9c67908ef8521c6e6ebadc5ad32053f70e5f72b6b8ab0aebf720eab6a4 WatchSource:0}: Error finding container e9d4ac9c67908ef8521c6e6ebadc5ad32053f70e5f72b6b8ab0aebf720eab6a4: Status 404 returned error can't find the container with id e9d4ac9c67908ef8521c6e6ebadc5ad32053f70e5f72b6b8ab0aebf720eab6a4 Nov 02 17:46:22.016166 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:22.016089 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ea104436-796f66c9bc-fdrgl"] Nov 02 17:46:22.228899 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:46:22.228870 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b56137_0532_4dff_bfbe_9ae561d67562.slice/crio-ab2c0300e7213a0b1dfeea8d9a844378624a7f0274d86e28ee3f1ef225856c93 WatchSource:0}: Error finding container ab2c0300e7213a0b1dfeea8d9a844378624a7f0274d86e28ee3f1ef225856c93: Status 404 returned error can't find the container with id ab2c0300e7213a0b1dfeea8d9a844378624a7f0274d86e28ee3f1ef225856c93 Nov 02 17:46:22.229403 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:22.229377 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ea104436-67cd4cf869-sk2pc"] Nov 02 17:46:22.311774 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:22.311740 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gg9fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 17:46:22.312044 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:22.312003 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" podUID="d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 17:46:22.394514 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:22.394477 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-k8n46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 17:46:22.394806 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:22.394741 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" podUID="cd908c8a-4026-4da6-95c5-34112ecc019a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 17:46:22.956406 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:22.954994 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ea104436-67cd4cf869-sk2pc" event={"ID":"b1b56137-0532-4dff-bfbe-9ae561d67562","Type":"ContainerStarted","Data":"def5f29ff75cd886d3b16754ef67f3532f70fc10eec6e84a78533188aa139924"} Nov 02 17:46:22.956406 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:22.955077 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ea104436-67cd4cf869-sk2pc" event={"ID":"b1b56137-0532-4dff-bfbe-9ae561d67562","Type":"ContainerStarted","Data":"ab2c0300e7213a0b1dfeea8d9a844378624a7f0274d86e28ee3f1ef225856c93"} Nov 02 17:46:22.956406 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:22.956346 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-ea104436-67cd4cf869-sk2pc" Nov 02 17:46:22.972233 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:22.972199 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea104436-796f66c9bc-fdrgl" event={"ID":"0891f7a9-f1df-49d5-b085-e2c343a05de5","Type":"ContainerStarted","Data":"89b533fdbe15f6ee9b9e0d51481887af5caa3b3911597be2b3f7d05adee47981"} Nov 02 17:46:22.980610 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:22.980578 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea104436-796f66c9bc-fdrgl" Nov 02 17:46:22.980803 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:22.980778 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea104436-796f66c9bc-fdrgl" event={"ID":"0891f7a9-f1df-49d5-b085-e2c343a05de5","Type":"ContainerStarted","Data":"e9d4ac9c67908ef8521c6e6ebadc5ad32053f70e5f72b6b8ab0aebf720eab6a4"} Nov 02 17:46:23.101853 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:23.101126 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-ea104436-67cd4cf869-sk2pc" podStartSLOduration=2.101092025 podStartE2EDuration="2.101092025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:23.028742444 +0000 UTC m=+1000.724097938" watchObservedRunningTime="2026-11-02 17:46:23.101092025 +0000 UTC m=+1000.796447499" Nov 02 17:46:23.226391 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:23.205507 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea104436-796f66c9bc-fdrgl" podStartSLOduration=2.205479826 podStartE2EDuration="2.205479826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:23.117711341 +0000 UTC m=+1000.813066834" watchObservedRunningTime="2026-11-02 17:46:23.205479826 +0000 UTC m=+1000.900835301" Nov 02 17:46:23.316015 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:23.315981 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gg9fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 17:46:23.316274 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:23.316245 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" podUID="d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 17:46:23.411839 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:23.407519 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-k8n46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 17:46:23.411839 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:23.407601 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" podUID="cd908c8a-4026-4da6-95c5-34112ecc019a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 17:46:24.026412 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.026209 4471 generic.go:356] "Generic (PLEG): container finished" podID="1c7aaa66-3d4a-4462-82d4-faf4ed4345f9" containerID="49c69119eb64eb39944e116cfaadca086eacb6203c20f4bb605867180932ce2b" exitCode=0 Nov 02 17:46:24.026988 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.026963 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" event={"ID":"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9","Type":"ContainerDied","Data":"49c69119eb64eb39944e116cfaadca086eacb6203c20f4bb605867180932ce2b"} Nov 02 17:46:24.027103 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.027011 4471 scope.go:122] "RemoveContainer" containerID="21a78147746a371b82dbfd9c3e52c5e48cab2a88aa7df734c4de840afc6c2022" Nov 02 17:46:24.039255 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.039234 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:46:24.144921 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.144839 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-credential-socket\") pod \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " Nov 02 17:46:24.145266 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.145020 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istio-podinfo\") pod \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " Nov 02 17:46:24.145325 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.145268 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istiod-ca-cert\") pod \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " Nov 02 17:46:24.145486 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.145455 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-workload-socket\") pod \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " Nov 02 17:46:24.145545 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.145530 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istio-token\" (UniqueName: \"kubernetes.io/projected/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istio-token\") pod \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " Nov 02 17:46:24.145642 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.145613 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-workload-certs\") pod \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " Nov 02 17:46:24.145700 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.145686 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istio-data\") pod \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " Nov 02 17:46:24.145811 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.145797 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-kube-api-access-xlpwf\" (UniqueName: \"kubernetes.io/projected/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-kube-api-access-xlpwf\") pod \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " Nov 02 17:46:24.145911 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.145897 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istio-envoy\") pod \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\" (UID: \"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9\") " Nov 02 17:46:24.146413 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.146382 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-credential-socket" pod "1c7aaa66-3d4a-4462-82d4-faf4ed4345f9" (UID: "1c7aaa66-3d4a-4462-82d4-faf4ed4345f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:46:24.146413 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.146390 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-workload-socket" pod "1c7aaa66-3d4a-4462-82d4-faf4ed4345f9" (UID: "1c7aaa66-3d4a-4462-82d4-faf4ed4345f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:46:24.146714 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.146689 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istiod-ca-cert" pod "1c7aaa66-3d4a-4462-82d4-faf4ed4345f9" (UID: "1c7aaa66-3d4a-4462-82d4-faf4ed4345f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:46:24.147187 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.147159 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istio-data" pod "1c7aaa66-3d4a-4462-82d4-faf4ed4345f9" (UID: "1c7aaa66-3d4a-4462-82d4-faf4ed4345f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:46:24.147391 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.147365 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:46:24.147494 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.147393 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:46:24.147494 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.147410 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:46:24.147494 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.147424 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:46:24.147999 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.147975 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-workload-certs" pod "1c7aaa66-3d4a-4462-82d4-faf4ed4345f9" (UID: "1c7aaa66-3d4a-4462-82d4-faf4ed4345f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:46:24.148710 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.148668 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istio-token" pod "1c7aaa66-3d4a-4462-82d4-faf4ed4345f9" (UID: "1c7aaa66-3d4a-4462-82d4-faf4ed4345f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:46:24.148907 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.148881 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istio-envoy" pod "1c7aaa66-3d4a-4462-82d4-faf4ed4345f9" (UID: "1c7aaa66-3d4a-4462-82d4-faf4ed4345f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:46:24.148984 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.148932 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istio-podinfo" pod "1c7aaa66-3d4a-4462-82d4-faf4ed4345f9" (UID: "1c7aaa66-3d4a-4462-82d4-faf4ed4345f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:46:24.149763 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.149745 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-kube-api-access-xlpwf" pod "1c7aaa66-3d4a-4462-82d4-faf4ed4345f9" (UID: "1c7aaa66-3d4a-4462-82d4-faf4ed4345f9"). InnerVolumeSpecName "kube-api-access-xlpwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:46:24.248608 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.248574 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:46:24.248608 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.248602 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:46:24.248608 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.248611 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlpwf\" (UniqueName: \"kubernetes.io/projected/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-kube-api-access-xlpwf\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:46:24.248837 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.248620 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:46:24.248837 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.248630 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:46:24.311949 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.311911 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gg9fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 17:46:24.312131 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.311960 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" podUID="d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 17:46:24.398390 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.396437 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-k8n46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 17:46:24.398390 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.396501 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" podUID="cd908c8a-4026-4da6-95c5-34112ecc019a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 17:46:24.499330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.490037 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-30a972ba-7867f88c69-7cbk4"] Nov 02 17:46:24.503329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.501382 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-30a972ba-899549479-jv7r7"] Nov 02 17:46:24.681617 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.672383 4471 patch_prober.go:29] interesting pod/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:46:24.681617 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.672446 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" podUID="1c7aaa66-3d4a-4462-82d4-faf4ed4345f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:24.725005 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.724967 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2"] Nov 02 17:46:24.725233 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:24.725198 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2" podUID="16e820b9-991c-4aaa-bdc3-1ba1076b5e81" containerName="maas-api" containerID="cri-o://b3ca280860c6aea185d00c7c8ac04d2967ba3cef32b71539bff266febdc0089c" gracePeriod=30 Nov 02 17:46:25.039351 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:25.039249 4471 generic.go:356] "Generic (PLEG): container finished" podID="16e820b9-991c-4aaa-bdc3-1ba1076b5e81" containerID="b3ca280860c6aea185d00c7c8ac04d2967ba3cef32b71539bff266febdc0089c" exitCode=0 Nov 02 17:46:25.039351 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:25.039380 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2" event={"ID":"16e820b9-991c-4aaa-bdc3-1ba1076b5e81","Type":"ContainerDied","Data":"b3ca280860c6aea185d00c7c8ac04d2967ba3cef32b71539bff266febdc0089c"} Nov 02 17:46:25.041801 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:25.041778 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" event={"ID":"1c7aaa66-3d4a-4462-82d4-faf4ed4345f9","Type":"ContainerDied","Data":"5449ba17dd9f249d5a9f441c4123d5fc833f2c60c8d6c593076d7dcb3c4439a1"} Nov 02 17:46:25.041971 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:25.041954 4471 scope.go:122] "RemoveContainer" containerID="49c69119eb64eb39944e116cfaadca086eacb6203c20f4bb605867180932ce2b" Nov 02 17:46:25.042158 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:25.041842 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx" Nov 02 17:46:25.237407 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:25.237126 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2" Nov 02 17:46:25.251330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:25.250355 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx"] Nov 02 17:46:25.296336 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:25.278284 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-556cd8a1-openshift-default-7576966799-mkkqx"] Nov 02 17:46:25.325327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:25.313857 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gg9fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 17:46:25.325327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:25.313919 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" podUID="d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 17:46:25.325327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:25.321609 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16e820b9-991c-4aaa-bdc3-1ba1076b5e81-metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e820b9-991c-4aaa-bdc3-1ba1076b5e81-metrics-certs\") pod \"16e820b9-991c-4aaa-bdc3-1ba1076b5e81\" (UID: \"16e820b9-991c-4aaa-bdc3-1ba1076b5e81\") " Nov 02 17:46:25.325327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:25.321705 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16e820b9-991c-4aaa-bdc3-1ba1076b5e81-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e820b9-991c-4aaa-bdc3-1ba1076b5e81-maas-api-tls\") pod \"16e820b9-991c-4aaa-bdc3-1ba1076b5e81\" (UID: \"16e820b9-991c-4aaa-bdc3-1ba1076b5e81\") " Nov 02 17:46:25.325327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:25.321982 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16e820b9-991c-4aaa-bdc3-1ba1076b5e81-kube-api-access-xdhcv\" (UniqueName: \"kubernetes.io/projected/16e820b9-991c-4aaa-bdc3-1ba1076b5e81-kube-api-access-xdhcv\") pod \"16e820b9-991c-4aaa-bdc3-1ba1076b5e81\" (UID: \"16e820b9-991c-4aaa-bdc3-1ba1076b5e81\") " Nov 02 17:46:25.325327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:25.322056 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16e820b9-991c-4aaa-bdc3-1ba1076b5e81-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e820b9-991c-4aaa-bdc3-1ba1076b5e81-openshift-service-ca\") pod \"16e820b9-991c-4aaa-bdc3-1ba1076b5e81\" (UID: \"16e820b9-991c-4aaa-bdc3-1ba1076b5e81\") " Nov 02 17:46:25.325327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:25.323340 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e820b9-991c-4aaa-bdc3-1ba1076b5e81-openshift-service-ca" pod "16e820b9-991c-4aaa-bdc3-1ba1076b5e81" (UID: "16e820b9-991c-4aaa-bdc3-1ba1076b5e81"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:46:25.400333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:25.391749 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-k8n46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 17:46:25.400333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:25.391817 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" podUID="cd908c8a-4026-4da6-95c5-34112ecc019a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 17:46:25.504432 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:25.443859 4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e820b9-991c-4aaa-bdc3-1ba1076b5e81-openshift-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:46:25.706886 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:25.704033 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16e820b9-991c-4aaa-bdc3-1ba1076b5e81-metrics-certs" pod "16e820b9-991c-4aaa-bdc3-1ba1076b5e81" (UID: "16e820b9-991c-4aaa-bdc3-1ba1076b5e81"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:46:25.839196 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:25.803730 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16e820b9-991c-4aaa-bdc3-1ba1076b5e81-maas-api-tls" pod "16e820b9-991c-4aaa-bdc3-1ba1076b5e81" (UID: "16e820b9-991c-4aaa-bdc3-1ba1076b5e81"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:46:25.839896 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:25.839827 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e820b9-991c-4aaa-bdc3-1ba1076b5e81-maas-api-tls\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:46:25.839896 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:25.839860 4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e820b9-991c-4aaa-bdc3-1ba1076b5e81-metrics-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:46:25.874986 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:25.874910 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e820b9-991c-4aaa-bdc3-1ba1076b5e81-kube-api-access-xdhcv" pod "16e820b9-991c-4aaa-bdc3-1ba1076b5e81" (UID: "16e820b9-991c-4aaa-bdc3-1ba1076b5e81"). InnerVolumeSpecName "kube-api-access-xdhcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:46:25.999330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:25.995942 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdhcv\" (UniqueName: \"kubernetes.io/projected/16e820b9-991c-4aaa-bdc3-1ba1076b5e81-kube-api-access-xdhcv\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:46:26.116277 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:26.116180 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2" event={"ID":"16e820b9-991c-4aaa-bdc3-1ba1076b5e81","Type":"ContainerDied","Data":"cac6b27b132d2a81f224a1d5724cb3daa820097b39155781fa5a30020e411cf2"} Nov 02 17:46:26.116497 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:26.116292 4471 scope.go:122] "RemoveContainer" containerID="b3ca280860c6aea185d00c7c8ac04d2967ba3cef32b71539bff266febdc0089c" Nov 02 17:46:26.116739 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:26.116715 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2" Nov 02 17:46:26.208851 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:26.208760 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2"] Nov 02 17:46:26.213007 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:26.212978 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-30a972ba-d47b755b4-6xvv2"] Nov 02 17:46:26.311481 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:26.311444 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gg9fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 17:46:26.311903 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:26.311495 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" podUID="d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 17:46:26.387384 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:26.386720 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-k8n46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 17:46:26.387384 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:26.386779 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" podUID="cd908c8a-4026-4da6-95c5-34112ecc019a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 17:46:26.444713 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:26.442903 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16e820b9-991c-4aaa-bdc3-1ba1076b5e81" path="/var/lib/kubelet/pods/16e820b9-991c-4aaa-bdc3-1ba1076b5e81/volumes" Nov 02 17:46:26.444713 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:26.443438 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c7aaa66-3d4a-4462-82d4-faf4ed4345f9" path="/var/lib/kubelet/pods/1c7aaa66-3d4a-4462-82d4-faf4ed4345f9/volumes" Nov 02 17:46:26.702020 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:26.701932 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-93321a4f-6fd4889ff5-r5gg6"] Nov 02 17:46:26.706330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:26.703238 4471 cpu_manager.go:414] "containerMap: removing container" podUID="16e820b9-991c-4aaa-bdc3-1ba1076b5e81" containerName="maas-api" Nov 02 17:46:26.706532 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:26.706512 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16e820b9-991c-4aaa-bdc3-1ba1076b5e81" containerName="maas-api" Nov 02 17:46:26.706656 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:26.706646 4471 cpu_manager.go:414] "containerMap: removing container" podUID="1c7aaa66-3d4a-4462-82d4-faf4ed4345f9" containerName="istio-proxy" Nov 02 17:46:26.706730 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:26.706721 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c7aaa66-3d4a-4462-82d4-faf4ed4345f9" containerName="istio-proxy" Nov 02 17:46:26.706814 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:26.706806 4471 cpu_manager.go:414] "containerMap: removing container" podUID="1c7aaa66-3d4a-4462-82d4-faf4ed4345f9" containerName="istio-proxy" Nov 02 17:46:26.706878 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:26.706870 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c7aaa66-3d4a-4462-82d4-faf4ed4345f9" containerName="istio-proxy" Nov 02 17:46:26.707166 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:26.707154 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="16e820b9-991c-4aaa-bdc3-1ba1076b5e81" containerName="maas-api" Nov 02 17:46:26.707988 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:26.707966 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c7aaa66-3d4a-4462-82d4-faf4ed4345f9" containerName="istio-proxy" Nov 02 17:46:26.708147 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:26.708132 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c7aaa66-3d4a-4462-82d4-faf4ed4345f9" containerName="istio-proxy" Nov 02 17:46:26.728423 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:26.723252 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-93321a4f-6fd4889ff5-r5gg6" Nov 02 17:46:26.792385 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:26.792350 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-93321a4f-dockercfg-57x47\"" Nov 02 17:46:26.793354 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:26.793327 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-93321a4f\"" Nov 02 17:46:26.835155 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:26.822377 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-93321a4f-6fd4889ff5-r5gg6"] Nov 02 17:46:26.845535 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:26.836278 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/748d8213-fb65-48ff-8280-fd153833f578-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-93321a4f-6fd4889ff5-r5gg6\" (UID: \"748d8213-fb65-48ff-8280-fd153833f578\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-93321a4f-6fd4889ff5-r5gg6" Nov 02 17:46:26.845535 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:26.836663 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vftg7\" (UniqueName: \"kubernetes.io/projected/748d8213-fb65-48ff-8280-fd153833f578-kube-api-access-vftg7\") pod \"payload-pre-processing-e2e-derive-93321a4f-6fd4889ff5-r5gg6\" (UID: \"748d8213-fb65-48ff-8280-fd153833f578\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-93321a4f-6fd4889ff5-r5gg6" Nov 02 17:46:26.951337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:26.938477 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/748d8213-fb65-48ff-8280-fd153833f578-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-93321a4f-6fd4889ff5-r5gg6\" (UID: \"748d8213-fb65-48ff-8280-fd153833f578\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-93321a4f-6fd4889ff5-r5gg6" Nov 02 17:46:26.951337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:26.938693 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vftg7\" (UniqueName: \"kubernetes.io/projected/748d8213-fb65-48ff-8280-fd153833f578-kube-api-access-vftg7\") pod \"payload-pre-processing-e2e-derive-93321a4f-6fd4889ff5-r5gg6\" (UID: \"748d8213-fb65-48ff-8280-fd153833f578\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-93321a4f-6fd4889ff5-r5gg6" Nov 02 17:46:26.951337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:26.939996 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/748d8213-fb65-48ff-8280-fd153833f578-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-93321a4f-6fd4889ff5-r5gg6\" (UID: \"748d8213-fb65-48ff-8280-fd153833f578\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-93321a4f-6fd4889ff5-r5gg6" Nov 02 17:46:27.079728 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:27.072832 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-93321a4f-684bd94557-hlncd"] Nov 02 17:46:27.163112 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:27.162444 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-93321a4f-684bd94557-hlncd" Nov 02 17:46:27.215483 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:27.215368 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-93321a4f-684bd94557-hlncd"] Nov 02 17:46:27.249533 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:27.245566 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vftg7\" (UniqueName: \"kubernetes.io/projected/748d8213-fb65-48ff-8280-fd153833f578-kube-api-access-vftg7\") pod \"payload-pre-processing-e2e-derive-93321a4f-6fd4889ff5-r5gg6\" (UID: \"748d8213-fb65-48ff-8280-fd153833f578\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-93321a4f-6fd4889ff5-r5gg6" Nov 02 17:46:27.302753 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:27.301887 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-93321a4f-6fd4889ff5-r5gg6" Nov 02 17:46:27.318973 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:27.318937 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gg9fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 17:46:27.319425 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:27.319178 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" podUID="d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 17:46:27.331633 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:27.331562 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgrt9\" (UniqueName: \"kubernetes.io/projected/62bbf303-13e9-4002-9236-253bd2d76062-kube-api-access-bgrt9\") pod \"payload-processing-e2e-derive-93321a4f-684bd94557-hlncd\" (UID: \"62bbf303-13e9-4002-9236-253bd2d76062\") " pod="openshift-ingress/payload-processing-e2e-derive-93321a4f-684bd94557-hlncd" Nov 02 17:46:27.332014 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:27.331994 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62bbf303-13e9-4002-9236-253bd2d76062-plugins-config-volume\") pod \"payload-processing-e2e-derive-93321a4f-684bd94557-hlncd\" (UID: \"62bbf303-13e9-4002-9236-253bd2d76062\") " pod="openshift-ingress/payload-processing-e2e-derive-93321a4f-684bd94557-hlncd" Nov 02 17:46:27.406131 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:27.406075 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-k8n46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 17:46:27.406419 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:27.406368 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" podUID="cd908c8a-4026-4da6-95c5-34112ecc019a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 17:46:27.434634 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:27.434606 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgrt9\" (UniqueName: \"kubernetes.io/projected/62bbf303-13e9-4002-9236-253bd2d76062-kube-api-access-bgrt9\") pod \"payload-processing-e2e-derive-93321a4f-684bd94557-hlncd\" (UID: \"62bbf303-13e9-4002-9236-253bd2d76062\") " pod="openshift-ingress/payload-processing-e2e-derive-93321a4f-684bd94557-hlncd" Nov 02 17:46:27.434818 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:27.434795 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62bbf303-13e9-4002-9236-253bd2d76062-plugins-config-volume\") pod \"payload-processing-e2e-derive-93321a4f-684bd94557-hlncd\" (UID: \"62bbf303-13e9-4002-9236-253bd2d76062\") " pod="openshift-ingress/payload-processing-e2e-derive-93321a4f-684bd94557-hlncd" Nov 02 17:46:27.435665 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:27.435644 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62bbf303-13e9-4002-9236-253bd2d76062-plugins-config-volume\") pod \"payload-processing-e2e-derive-93321a4f-684bd94557-hlncd\" (UID: \"62bbf303-13e9-4002-9236-253bd2d76062\") " pod="openshift-ingress/payload-processing-e2e-derive-93321a4f-684bd94557-hlncd" Nov 02 17:46:27.514157 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:27.514098 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgrt9\" (UniqueName: \"kubernetes.io/projected/62bbf303-13e9-4002-9236-253bd2d76062-kube-api-access-bgrt9\") pod \"payload-processing-e2e-derive-93321a4f-684bd94557-hlncd\" (UID: \"62bbf303-13e9-4002-9236-253bd2d76062\") " pod="openshift-ingress/payload-processing-e2e-derive-93321a4f-684bd94557-hlncd" Nov 02 17:46:27.634624 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:27.634541 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-93321a4f-684bd94557-hlncd" Nov 02 17:46:27.787072 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:27.775260 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-93321a4f-6fd4889ff5-r5gg6"] Nov 02 17:46:28.220258 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:28.220163 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-93321a4f-684bd94557-hlncd"] Nov 02 17:46:28.236846 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:28.235292 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-93321a4f-6fd4889ff5-r5gg6" event={"ID":"748d8213-fb65-48ff-8280-fd153833f578","Type":"ContainerStarted","Data":"efcfdda3160a0ca4db9eef37a139962bd88c451c222e57e99965cb1d9a289744"} Nov 02 17:46:28.236846 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:28.235378 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-93321a4f-6fd4889ff5-r5gg6" event={"ID":"748d8213-fb65-48ff-8280-fd153833f578","Type":"ContainerStarted","Data":"999134bfee2f150b0b3b08374ee37a2f3254aff67febae834beb33792a4eb58a"} Nov 02 17:46:28.236846 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:28.236762 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-93321a4f-6fd4889ff5-r5gg6" Nov 02 17:46:28.333728 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:28.333689 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gg9fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 17:46:28.334501 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:28.334458 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" podUID="d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 17:46:28.344960 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:28.303287 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-93321a4f-6fd4889ff5-r5gg6" podStartSLOduration=2.30326308 podStartE2EDuration="2.30326308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:28.290755015 +0000 UTC m=+1005.986110501" watchObservedRunningTime="2026-11-02 17:46:28.30326308 +0000 UTC m=+1005.998618557" Nov 02 17:46:28.420027 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:28.419478 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-k8n46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 17:46:28.420027 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:28.419545 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" podUID="cd908c8a-4026-4da6-95c5-34112ecc019a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 17:46:29.314331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:29.299233 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-93321a4f-684bd94557-hlncd" event={"ID":"62bbf303-13e9-4002-9236-253bd2d76062","Type":"ContainerStarted","Data":"33064384c433ff89315d37528c2c7412b6e6eb89cd80fbb5d8e52f17df6d8777"} Nov 02 17:46:29.314331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:29.299359 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-93321a4f-684bd94557-hlncd" event={"ID":"62bbf303-13e9-4002-9236-253bd2d76062","Type":"ContainerStarted","Data":"57bdcfdf31eea48bc0dec778c41fd62ab7a7b3f3cd2a92993d0ab051057de1dc"} Nov 02 17:46:29.314331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:29.299412 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-93321a4f-684bd94557-hlncd" Nov 02 17:46:29.314331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:29.312698 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gg9fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 17:46:29.314331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:29.312760 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" podUID="d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 17:46:29.363003 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:29.362917 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-93321a4f-684bd94557-hlncd" podStartSLOduration=3.362897995 podStartE2EDuration="3.362897995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:29.360978078 +0000 UTC m=+1007.056333552" watchObservedRunningTime="2026-11-02 17:46:29.362897995 +0000 UTC m=+1007.058253468" Nov 02 17:46:29.386753 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:29.386709 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-k8n46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 17:46:29.386984 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:29.386779 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" podUID="cd908c8a-4026-4da6-95c5-34112ecc019a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 17:46:29.514439 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:29.514386 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-30a972ba-899549479-jv7r7" podUID="3b4ac904-c840-4caa-847b-e77250eaef2d" containerName="payload-pre-processing" containerID="cri-o://ff7b58bf9b813b87cb4e588c18a7583c6a781687f4f28fbeaf7ac90de8482c6d" gracePeriod=25 Nov 02 17:46:29.514883 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:29.514848 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-30a972ba-7867f88c69-7cbk4" podUID="5ea7040b-14c0-4b5c-ad13-0a77e8026bd9" containerName="payload-processing" containerID="cri-o://c32a060e1edd098a5db29e495577d232ea840f4b6315465d0f542abdf1f84e9a" gracePeriod=25 Nov 02 17:46:30.062369 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.058656 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-30a972ba-7867f88c69-7cbk4" Nov 02 17:46:30.137566 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.137525 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-30a972ba-899549479-jv7r7" Nov 02 17:46:30.212981 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.212942 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ea7040b-14c0-4b5c-ad13-0a77e8026bd9-kube-api-access-dpn4s\" (UniqueName: \"kubernetes.io/projected/5ea7040b-14c0-4b5c-ad13-0a77e8026bd9-kube-api-access-dpn4s\") pod \"5ea7040b-14c0-4b5c-ad13-0a77e8026bd9\" (UID: \"5ea7040b-14c0-4b5c-ad13-0a77e8026bd9\") " Nov 02 17:46:30.213172 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.213004 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ea7040b-14c0-4b5c-ad13-0a77e8026bd9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ea7040b-14c0-4b5c-ad13-0a77e8026bd9-plugins-config-volume\") pod \"5ea7040b-14c0-4b5c-ad13-0a77e8026bd9\" (UID: \"5ea7040b-14c0-4b5c-ad13-0a77e8026bd9\") " Nov 02 17:46:30.213172 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.213166 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b4ac904-c840-4caa-847b-e77250eaef2d-kube-api-access-kpphk\" (UniqueName: \"kubernetes.io/projected/3b4ac904-c840-4caa-847b-e77250eaef2d-kube-api-access-kpphk\") pod \"3b4ac904-c840-4caa-847b-e77250eaef2d\" (UID: \"3b4ac904-c840-4caa-847b-e77250eaef2d\") " Nov 02 17:46:30.213286 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.213221 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b4ac904-c840-4caa-847b-e77250eaef2d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b4ac904-c840-4caa-847b-e77250eaef2d-plugins-config-volume\") pod \"3b4ac904-c840-4caa-847b-e77250eaef2d\" (UID: \"3b4ac904-c840-4caa-847b-e77250eaef2d\") " Nov 02 17:46:30.214562 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.214383 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ea7040b-14c0-4b5c-ad13-0a77e8026bd9-plugins-config-volume" pod "5ea7040b-14c0-4b5c-ad13-0a77e8026bd9" (UID: "5ea7040b-14c0-4b5c-ad13-0a77e8026bd9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:46:30.215858 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.215829 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b4ac904-c840-4caa-847b-e77250eaef2d-plugins-config-volume" pod "3b4ac904-c840-4caa-847b-e77250eaef2d" (UID: "3b4ac904-c840-4caa-847b-e77250eaef2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:46:30.216363 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.216196 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b4ac904-c840-4caa-847b-e77250eaef2d-kube-api-access-kpphk" pod "3b4ac904-c840-4caa-847b-e77250eaef2d" (UID: "3b4ac904-c840-4caa-847b-e77250eaef2d"). InnerVolumeSpecName "kube-api-access-kpphk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:46:30.216677 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.216653 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea7040b-14c0-4b5c-ad13-0a77e8026bd9-kube-api-access-dpn4s" pod "5ea7040b-14c0-4b5c-ad13-0a77e8026bd9" (UID: "5ea7040b-14c0-4b5c-ad13-0a77e8026bd9"). InnerVolumeSpecName "kube-api-access-dpn4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:46:30.307200 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.307164 4471 generic.go:356] "Generic (PLEG): container finished" podID="5ea7040b-14c0-4b5c-ad13-0a77e8026bd9" containerID="c32a060e1edd098a5db29e495577d232ea840f4b6315465d0f542abdf1f84e9a" exitCode=0 Nov 02 17:46:30.307460 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.307211 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-30a972ba-7867f88c69-7cbk4" Nov 02 17:46:30.307460 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.307252 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-30a972ba-7867f88c69-7cbk4" event={"ID":"5ea7040b-14c0-4b5c-ad13-0a77e8026bd9","Type":"ContainerDied","Data":"c32a060e1edd098a5db29e495577d232ea840f4b6315465d0f542abdf1f84e9a"} Nov 02 17:46:30.307460 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.307352 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-30a972ba-7867f88c69-7cbk4" event={"ID":"5ea7040b-14c0-4b5c-ad13-0a77e8026bd9","Type":"ContainerDied","Data":"5cb4e3d0afc1784cc8fad42c69ceebe0f55c69838b7ce9e9751ee14f18d22f9f"} Nov 02 17:46:30.307460 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.307390 4471 scope.go:122] "RemoveContainer" containerID="c32a060e1edd098a5db29e495577d232ea840f4b6315465d0f542abdf1f84e9a" Nov 02 17:46:30.308820 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.308794 4471 generic.go:356] "Generic (PLEG): container finished" podID="3b4ac904-c840-4caa-847b-e77250eaef2d" containerID="ff7b58bf9b813b87cb4e588c18a7583c6a781687f4f28fbeaf7ac90de8482c6d" exitCode=0 Nov 02 17:46:30.310043 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.310020 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-30a972ba-899549479-jv7r7" Nov 02 17:46:30.314868 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.314478 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-30a972ba-899549479-jv7r7" event={"ID":"3b4ac904-c840-4caa-847b-e77250eaef2d","Type":"ContainerDied","Data":"ff7b58bf9b813b87cb4e588c18a7583c6a781687f4f28fbeaf7ac90de8482c6d"} Nov 02 17:46:30.314868 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.314534 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-30a972ba-899549479-jv7r7" event={"ID":"3b4ac904-c840-4caa-847b-e77250eaef2d","Type":"ContainerDied","Data":"fc15e911a78da8aec81db6b95d71af16fe82d8359a704d78f86e181f618b19e7"} Nov 02 17:46:30.317184 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.315631 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gg9fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 17:46:30.317184 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.315688 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" podUID="d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 17:46:30.317184 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.316176 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpphk\" (UniqueName: \"kubernetes.io/projected/3b4ac904-c840-4caa-847b-e77250eaef2d-kube-api-access-kpphk\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:46:30.317184 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.316204 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b4ac904-c840-4caa-847b-e77250eaef2d-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:46:30.317184 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.316219 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpn4s\" (UniqueName: \"kubernetes.io/projected/5ea7040b-14c0-4b5c-ad13-0a77e8026bd9-kube-api-access-dpn4s\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:46:30.317184 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.316232 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ea7040b-14c0-4b5c-ad13-0a77e8026bd9-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:46:30.321469 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.321445 4471 scope.go:122] "RemoveContainer" containerID="c32a060e1edd098a5db29e495577d232ea840f4b6315465d0f542abdf1f84e9a" Nov 02 17:46:30.321919 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:46:30.321775 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c32a060e1edd098a5db29e495577d232ea840f4b6315465d0f542abdf1f84e9a\": container with ID starting with c32a060e1edd098a5db29e495577d232ea840f4b6315465d0f542abdf1f84e9a not found: ID does not exist" containerID="c32a060e1edd098a5db29e495577d232ea840f4b6315465d0f542abdf1f84e9a" Nov 02 17:46:30.321919 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.321809 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c32a060e1edd098a5db29e495577d232ea840f4b6315465d0f542abdf1f84e9a"} err="failed to get container status \"c32a060e1edd098a5db29e495577d232ea840f4b6315465d0f542abdf1f84e9a\": rpc error: code = NotFound desc = could not find container \"c32a060e1edd098a5db29e495577d232ea840f4b6315465d0f542abdf1f84e9a\": container with ID starting with c32a060e1edd098a5db29e495577d232ea840f4b6315465d0f542abdf1f84e9a not found: ID does not exist" Nov 02 17:46:30.321919 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.321828 4471 scope.go:122] "RemoveContainer" containerID="ff7b58bf9b813b87cb4e588c18a7583c6a781687f4f28fbeaf7ac90de8482c6d" Nov 02 17:46:30.336487 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.336452 4471 scope.go:122] "RemoveContainer" containerID="ff7b58bf9b813b87cb4e588c18a7583c6a781687f4f28fbeaf7ac90de8482c6d" Nov 02 17:46:30.336868 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:46:30.336841 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff7b58bf9b813b87cb4e588c18a7583c6a781687f4f28fbeaf7ac90de8482c6d\": container with ID starting with ff7b58bf9b813b87cb4e588c18a7583c6a781687f4f28fbeaf7ac90de8482c6d not found: ID does not exist" containerID="ff7b58bf9b813b87cb4e588c18a7583c6a781687f4f28fbeaf7ac90de8482c6d" Nov 02 17:46:30.336977 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.336878 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff7b58bf9b813b87cb4e588c18a7583c6a781687f4f28fbeaf7ac90de8482c6d"} err="failed to get container status \"ff7b58bf9b813b87cb4e588c18a7583c6a781687f4f28fbeaf7ac90de8482c6d\": rpc error: code = NotFound desc = could not find container \"ff7b58bf9b813b87cb4e588c18a7583c6a781687f4f28fbeaf7ac90de8482c6d\": container with ID starting with ff7b58bf9b813b87cb4e588c18a7583c6a781687f4f28fbeaf7ac90de8482c6d not found: ID does not exist" Nov 02 17:46:30.375323 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.375268 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-30a972ba-899549479-jv7r7"] Nov 02 17:46:30.384574 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.384547 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-30a972ba-899549479-jv7r7"] Nov 02 17:46:30.386618 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.386593 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-k8n46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 17:46:30.386721 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.386642 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" podUID="cd908c8a-4026-4da6-95c5-34112ecc019a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 17:46:30.452257 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.452225 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b4ac904-c840-4caa-847b-e77250eaef2d" path="/var/lib/kubelet/pods/3b4ac904-c840-4caa-847b-e77250eaef2d/volumes" Nov 02 17:46:30.452589 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.452574 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-30a972ba-7867f88c69-7cbk4"] Nov 02 17:46:30.454407 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:30.454249 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-30a972ba-7867f88c69-7cbk4"] Nov 02 17:46:31.178808 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.178776 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t"] Nov 02 17:46:31.180124 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.180093 4471 cpu_manager.go:414] "containerMap: removing container" podUID="3b4ac904-c840-4caa-847b-e77250eaef2d" containerName="payload-pre-processing" Nov 02 17:46:31.180294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.180279 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b4ac904-c840-4caa-847b-e77250eaef2d" containerName="payload-pre-processing" Nov 02 17:46:31.180444 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.180430 4471 cpu_manager.go:414] "containerMap: removing container" podUID="5ea7040b-14c0-4b5c-ad13-0a77e8026bd9" containerName="payload-processing" Nov 02 17:46:31.184395 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.184370 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ea7040b-14c0-4b5c-ad13-0a77e8026bd9" containerName="payload-processing" Nov 02 17:46:31.184748 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.184729 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b4ac904-c840-4caa-847b-e77250eaef2d" containerName="payload-pre-processing" Nov 02 17:46:31.197949 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.197920 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ea7040b-14c0-4b5c-ad13-0a77e8026bd9" containerName="payload-processing" Nov 02 17:46:31.224377 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.208497 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:46:31.289331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.261032 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-b91583-openshift-default-dockercfg-68cp5\"" Nov 02 17:46:31.335382 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.314907 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gg9fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 17:46:31.335382 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.314967 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" podUID="d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 17:46:31.385334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.376750 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-credential-socket\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:46:31.385334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.376897 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istiod-ca-cert\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:46:31.385334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.376962 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cfpk\" (UniqueName: \"kubernetes.io/projected/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-kube-api-access-2cfpk\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:46:31.385334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.377014 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istio-token\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:46:31.385334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.377165 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istio-envoy\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:46:31.385334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.377250 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istio-podinfo\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:46:31.385334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.377281 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istio-data\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:46:31.385334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.377341 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-workload-certs\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:46:31.385334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.377379 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-workload-socket\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:46:31.392721 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.392675 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-k8n46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 17:46:31.393229 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.392741 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" podUID="cd908c8a-4026-4da6-95c5-34112ecc019a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 17:46:31.394653 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.394614 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t"] Nov 02 17:46:31.498479 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.498178 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-credential-socket\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:46:31.498479 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.498339 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istiod-ca-cert\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:46:31.498479 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.498404 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cfpk\" (UniqueName: \"kubernetes.io/projected/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-kube-api-access-2cfpk\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:46:31.498479 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.498457 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istio-token\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:46:31.498818 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.498604 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istio-envoy\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:46:31.498818 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.498685 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istio-podinfo\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:46:31.504424 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.501148 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istiod-ca-cert\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:46:31.533226 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.531847 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-credential-socket\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:46:31.543796 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.543757 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istio-data\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:46:31.553433 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.553398 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-workload-certs\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:46:31.553661 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.553645 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-workload-socket\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:46:31.554352 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.551811 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istio-data\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:46:31.554749 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.554729 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-workload-certs\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:46:31.555107 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.555089 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-workload-socket\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:46:31.592270 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.574112 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istio-token\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:46:31.614033 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.611948 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istio-envoy\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:46:31.621742 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.621708 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istio-podinfo\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:46:31.632231 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.632197 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cfpk\" (UniqueName: \"kubernetes.io/projected/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-kube-api-access-2cfpk\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:46:31.641810 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:31.641770 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:46:32.320753 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:32.320194 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gg9fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 17:46:32.320753 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:32.320275 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" podUID="d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 17:46:32.521175 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:32.521143 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ea7040b-14c0-4b5c-ad13-0a77e8026bd9" path="/var/lib/kubelet/pods/5ea7040b-14c0-4b5c-ad13-0a77e8026bd9/volumes" Nov 02 17:46:32.591992 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:32.591291 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t"] Nov 02 17:46:32.630331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:32.616552 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:46:32.630331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:32.621149 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:46:32.630331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:32.621203 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:46:33.437442 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:33.437405 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-k8n46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:46:33.437704 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:33.437671 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" podUID="cd908c8a-4026-4da6-95c5-34112ecc019a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:33.544375 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:33.543118 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" event={"ID":"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c","Type":"ContainerStarted","Data":"ab033f6cbb7efeb30c7490fd232a38b8081b46fcd5aa84659a37c98af3341d44"} Nov 02 17:46:33.544375 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:33.543189 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" event={"ID":"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c","Type":"ContainerStarted","Data":"4d1317b69ce028c87cb9f8b80146e05a9eac5bc42503ac4fee06bf1ca5d27cd9"} Nov 02 17:46:33.630516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:33.630453 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" podStartSLOduration=2.6304341300000003 podStartE2EDuration="2.63043413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:33.629011664 +0000 UTC m=+1011.324367151" watchObservedRunningTime="2026-11-02 17:46:33.63043413 +0000 UTC m=+1011.325789605" Nov 02 17:46:33.648338 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:33.645115 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:46:33.657355 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:33.656731 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 17:46:33.661329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:33.658321 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 17:46:33.661329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:33.658534 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:46:33.661329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:33.660893 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 17:46:34.123155 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:34.122377 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea104436-796f66c9bc-fdrgl" Nov 02 17:46:34.361359 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:34.356490 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gg9fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:46:34.361359 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:34.356573 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" podUID="d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:34.363673 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:34.363638 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 17:46:34.365839 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:34.365594 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 17:46:34.372944 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:34.370260 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 17:46:34.583703 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:34.583672 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:46:34.613745 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:34.613707 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:46:35.045550 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:35.045519 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-ea104436-67cd4cf869-sk2pc" Nov 02 17:46:38.423178 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:38.423135 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b91583-66bb7b5959-5596q"] Nov 02 17:46:38.426981 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:38.426961 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b91583-66bb7b5959-5596q" Nov 02 17:46:38.450507 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:38.450480 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b91583-66bb7b5959-5596q"] Nov 02 17:46:38.459726 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:38.459697 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-b91583\"" Nov 02 17:46:38.459880 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:38.459821 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-b91583-dockercfg-vbwks\"" Nov 02 17:46:38.585330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:38.567669 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d94a6978-2092-4c1d-8731-f18298e5369b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-b91583-66bb7b5959-5596q\" (UID: \"d94a6978-2092-4c1d-8731-f18298e5369b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b91583-66bb7b5959-5596q" Nov 02 17:46:38.609325 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:38.593878 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfhsw\" (UniqueName: \"kubernetes.io/projected/d94a6978-2092-4c1d-8731-f18298e5369b-kube-api-access-lfhsw\") pod \"payload-pre-processing-e2e-shared-a-w6-b91583-66bb7b5959-5596q\" (UID: \"d94a6978-2092-4c1d-8731-f18298e5369b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b91583-66bb7b5959-5596q" Nov 02 17:46:38.748410 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:38.723697 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d94a6978-2092-4c1d-8731-f18298e5369b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-b91583-66bb7b5959-5596q\" (UID: \"d94a6978-2092-4c1d-8731-f18298e5369b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b91583-66bb7b5959-5596q" Nov 02 17:46:38.776763 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:38.776723 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfhsw\" (UniqueName: \"kubernetes.io/projected/d94a6978-2092-4c1d-8731-f18298e5369b-kube-api-access-lfhsw\") pod \"payload-pre-processing-e2e-shared-a-w6-b91583-66bb7b5959-5596q\" (UID: \"d94a6978-2092-4c1d-8731-f18298e5369b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b91583-66bb7b5959-5596q" Nov 02 17:46:38.790609 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:38.726798 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d94a6978-2092-4c1d-8731-f18298e5369b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-b91583-66bb7b5959-5596q\" (UID: \"d94a6978-2092-4c1d-8731-f18298e5369b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b91583-66bb7b5959-5596q" Nov 02 17:46:38.844382 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:38.844347 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfhsw\" (UniqueName: \"kubernetes.io/projected/d94a6978-2092-4c1d-8731-f18298e5369b-kube-api-access-lfhsw\") pod \"payload-pre-processing-e2e-shared-a-w6-b91583-66bb7b5959-5596q\" (UID: \"d94a6978-2092-4c1d-8731-f18298e5369b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b91583-66bb7b5959-5596q" Nov 02 17:46:39.041588 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:39.041556 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b91583-66bb7b5959-5596q" Nov 02 17:46:39.284498 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:46:39.284440 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:46:39.345112 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:46:39.345078 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd94a6978_2092_4c1d_8731_f18298e5369b.slice/crio-3400bccd90e613db0c114d623b4671722f4470e524aaf78489949985cf82cc6f WatchSource:0}: Error finding container 3400bccd90e613db0c114d623b4671722f4470e524aaf78489949985cf82cc6f: Status 404 returned error can't find the container with id 3400bccd90e613db0c114d623b4671722f4470e524aaf78489949985cf82cc6f Nov 02 17:46:39.347294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:39.347266 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b91583-66bb7b5959-5596q"] Nov 02 17:46:39.850342 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:39.849638 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b91583-66bb7b5959-5596q" event={"ID":"d94a6978-2092-4c1d-8731-f18298e5369b","Type":"ContainerStarted","Data":"f90597617984abc94cdd86d1496200ac9b3df4ad6e241b3bde6127aa9dd3ebc8"} Nov 02 17:46:39.850342 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:39.849707 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b91583-66bb7b5959-5596q" event={"ID":"d94a6978-2092-4c1d-8731-f18298e5369b","Type":"ContainerStarted","Data":"3400bccd90e613db0c114d623b4671722f4470e524aaf78489949985cf82cc6f"} Nov 02 17:46:39.852794 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:39.852508 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b91583-66bb7b5959-5596q" Nov 02 17:46:39.899943 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:39.899890 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b91583-66bb7b5959-5596q" podStartSLOduration=1.899875358 podStartE2EDuration="1.899875358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:39.897753862 +0000 UTC m=+1017.593109356" watchObservedRunningTime="2026-11-02 17:46:39.899875358 +0000 UTC m=+1017.595230867" Nov 02 17:46:40.368346 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:40.367277 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-93321a4f-6fd4889ff5-r5gg6" Nov 02 17:46:41.409493 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:41.409461 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-93321a4f-684bd94557-hlncd" Nov 02 17:46:51.868474 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:51.868443 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b91583-66bb7b5959-5596q" Nov 02 17:46:54.587294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:54.587261 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg"] Nov 02 17:46:54.591689 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:54.591668 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg" Nov 02 17:46:54.624573 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:54.624537 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-b91583-serving-cert\"" Nov 02 17:46:54.630050 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:54.630024 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg"] Nov 02 17:46:54.654467 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:54.654436 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db82dff8-26f4-43f9-a544-aa605d6f5ac4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg\" (UID: \"db82dff8-26f4-43f9-a544-aa605d6f5ac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg" Nov 02 17:46:54.654734 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:54.654718 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps4n4\" (UniqueName: \"kubernetes.io/projected/db82dff8-26f4-43f9-a544-aa605d6f5ac4-kube-api-access-ps4n4\") pod \"maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg\" (UID: \"db82dff8-26f4-43f9-a544-aa605d6f5ac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg" Nov 02 17:46:54.654795 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:54.654782 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db82dff8-26f4-43f9-a544-aa605d6f5ac4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg\" (UID: \"db82dff8-26f4-43f9-a544-aa605d6f5ac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg" Nov 02 17:46:54.654849 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:54.654829 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db82dff8-26f4-43f9-a544-aa605d6f5ac4-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg\" (UID: \"db82dff8-26f4-43f9-a544-aa605d6f5ac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg" Nov 02 17:46:54.756060 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:54.756024 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps4n4\" (UniqueName: \"kubernetes.io/projected/db82dff8-26f4-43f9-a544-aa605d6f5ac4-kube-api-access-ps4n4\") pod \"maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg\" (UID: \"db82dff8-26f4-43f9-a544-aa605d6f5ac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg" Nov 02 17:46:54.756266 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:54.756123 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db82dff8-26f4-43f9-a544-aa605d6f5ac4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg\" (UID: \"db82dff8-26f4-43f9-a544-aa605d6f5ac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg" Nov 02 17:46:54.756266 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:54.756151 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db82dff8-26f4-43f9-a544-aa605d6f5ac4-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg\" (UID: \"db82dff8-26f4-43f9-a544-aa605d6f5ac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg" Nov 02 17:46:54.756450 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:46:54.756277 4471 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b91583-serving-cert: secret "maas-api-e2e-shared-a-w6-b91583-serving-cert" not found Nov 02 17:46:54.756450 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:54.756375 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db82dff8-26f4-43f9-a544-aa605d6f5ac4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg\" (UID: \"db82dff8-26f4-43f9-a544-aa605d6f5ac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg" Nov 02 17:46:54.756450 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:46:54.756407 4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db82dff8-26f4-43f9-a544-aa605d6f5ac4-maas-api-tls podName:db82dff8-26f4-43f9-a544-aa605d6f5ac4 nodeName:}" failed. No retries permitted until 2026-11-02 17:46:55.256383486 +0000 UTC m=+1032.951738952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/db82dff8-26f4-43f9-a544-aa605d6f5ac4-maas-api-tls") pod "maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg" (UID: "db82dff8-26f4-43f9-a544-aa605d6f5ac4") : secret "maas-api-e2e-shared-a-w6-b91583-serving-cert" not found Nov 02 17:46:54.757429 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:54.757403 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db82dff8-26f4-43f9-a544-aa605d6f5ac4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg\" (UID: \"db82dff8-26f4-43f9-a544-aa605d6f5ac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg" Nov 02 17:46:54.759670 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:54.759649 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db82dff8-26f4-43f9-a544-aa605d6f5ac4-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg\" (UID: \"db82dff8-26f4-43f9-a544-aa605d6f5ac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg" Nov 02 17:46:54.782845 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:54.782815 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps4n4\" (UniqueName: \"kubernetes.io/projected/db82dff8-26f4-43f9-a544-aa605d6f5ac4-kube-api-access-ps4n4\") pod \"maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg\" (UID: \"db82dff8-26f4-43f9-a544-aa605d6f5ac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg" Nov 02 17:46:55.265088 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:55.265056 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db82dff8-26f4-43f9-a544-aa605d6f5ac4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg\" (UID: \"db82dff8-26f4-43f9-a544-aa605d6f5ac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg" Nov 02 17:46:55.267520 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:55.267498 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db82dff8-26f4-43f9-a544-aa605d6f5ac4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg\" (UID: \"db82dff8-26f4-43f9-a544-aa605d6f5ac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg" Nov 02 17:46:55.506479 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:55.506391 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg" Nov 02 17:46:56.021182 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:56.020170 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg"] Nov 02 17:46:56.139726 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:56.139637 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg" event={"ID":"db82dff8-26f4-43f9-a544-aa605d6f5ac4","Type":"ContainerStarted","Data":"9caf0fcf659457dd25e3609f416077a64830ac0296506dc9d10463b114eefb12"} Nov 02 17:46:57.148685 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:57.148648 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg" event={"ID":"db82dff8-26f4-43f9-a544-aa605d6f5ac4","Type":"ContainerStarted","Data":"e1fd9a0f9f3a3bbaaa2c67ac9149b0569efd251315c6597f96f43fd29ffdcea0"} Nov 02 17:46:57.149067 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:57.148786 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg" Nov 02 17:46:57.245905 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:46:57.245846 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg" podStartSLOduration=2.807543723 podStartE2EDuration="3.245829476s" totalImagesPullingTime="438.285753ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:57.242214191 +0000 UTC m=+1034.937569665" watchObservedRunningTime="2026-11-02 17:46:57.245829476 +0000 UTC m=+1034.941185007" Nov 02 17:47:03.162600 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:03.162570 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg" Nov 02 17:47:09.184125 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:47:09.184085 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:47:09.931451 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:09.931412 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:47:16.214045 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:16.214004 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ea104436-67cd4cf869-sk2pc"] Nov 02 17:47:16.265374 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:16.255477 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ea104436-796f66c9bc-fdrgl"] Nov 02 17:47:18.890762 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:18.890719 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-93321a4f-684bd94557-hlncd"] Nov 02 17:47:18.926506 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:18.926469 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-93321a4f-6fd4889ff5-r5gg6"] Nov 02 17:47:19.392369 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:19.356158 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5"] Nov 02 17:47:19.640336 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:19.613034 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5"] Nov 02 17:47:19.640336 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:19.615476 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" Nov 02 17:47:19.750452 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:19.697501 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istio-podinfo\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" Nov 02 17:47:19.837536 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:19.826412 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istiod-ca-cert\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" Nov 02 17:47:19.837536 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:19.826546 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-workload-socket\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" Nov 02 17:47:19.837536 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:19.826590 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istio-token\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" Nov 02 17:47:19.837536 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:19.826657 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istio-data\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" Nov 02 17:47:19.837536 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:19.826937 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-credential-socket\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" Nov 02 17:47:19.837536 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:19.826998 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istio-envoy\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" Nov 02 17:47:19.837536 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:19.827040 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-workload-certs\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" Nov 02 17:47:19.837536 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:19.827133 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmpsr\" (UniqueName: \"kubernetes.io/projected/1a1b8c9e-6e63-4397-b3c0-46055331baa5-kube-api-access-kmpsr\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" Nov 02 17:47:19.930544 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:19.930143 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmpsr\" (UniqueName: \"kubernetes.io/projected/1a1b8c9e-6e63-4397-b3c0-46055331baa5-kube-api-access-kmpsr\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" Nov 02 17:47:19.930544 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:19.930367 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istio-podinfo\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" Nov 02 17:47:19.930544 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:19.930417 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istiod-ca-cert\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" Nov 02 17:47:19.930544 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:19.930487 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-workload-socket\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" Nov 02 17:47:19.930544 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:19.930528 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istio-token\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" Nov 02 17:47:19.931265 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:19.930575 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istio-data\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" Nov 02 17:47:19.931265 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:19.930700 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-credential-socket\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" Nov 02 17:47:19.931265 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:19.930747 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istio-envoy\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" Nov 02 17:47:19.931265 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:19.930789 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-workload-certs\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" Nov 02 17:47:19.937419 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:19.936874 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-workload-socket\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" Nov 02 17:47:19.946855 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:19.946783 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istio-data\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" Nov 02 17:47:19.948753 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:19.948064 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istiod-ca-cert\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" Nov 02 17:47:19.948923 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:19.948380 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-credential-socket\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" Nov 02 17:47:19.972334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:19.969104 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-workload-certs\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" Nov 02 17:47:20.011846 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:20.011365 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmpsr\" (UniqueName: \"kubernetes.io/projected/1a1b8c9e-6e63-4397-b3c0-46055331baa5-kube-api-access-kmpsr\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" Nov 02 17:47:20.012018 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:20.011896 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istio-token\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" Nov 02 17:47:20.012755 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:20.012728 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istio-podinfo\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" Nov 02 17:47:20.049109 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:20.049071 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istio-envoy\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" Nov 02 17:47:20.068899 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:20.068620 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" Nov 02 17:47:20.241335 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:20.168956 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt"] Nov 02 17:47:20.247798 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:20.247452 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" podUID="613f1277-5dc5-422e-8b92-d9be8ac858c4" containerName="istio-proxy" containerID="cri-o://053c1cc3d43c4bb595a4e790118314ee2260d473843fe0590ff297f6845112d7" gracePeriod=30 Nov 02 17:47:20.529931 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:20.529885 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:47:20.530489 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:20.530454 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 02 17:47:20.555368 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:20.542123 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:47:20.587331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:20.571029 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:47:20.733339 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:20.578492 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:47:20.757058 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:20.643782 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:47:20.757058 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:20.737007 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" podUID="1ff9e12a-8b9c-4556-ae7a-fa3011dd516c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:47:21.214465 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:21.214369 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-ea104436-67cd4cf869-sk2pc" podUID="b1b56137-0532-4dff-bfbe-9ae561d67562" containerName="payload-processing" containerID="cri-o://def5f29ff75cd886d3b16754ef67f3532f70fc10eec6e84a78533188aa139924" gracePeriod=25 Nov 02 17:47:21.238487 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:47:21.238436 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a1b8c9e_6e63_4397_b3c0_46055331baa5.slice/crio-dd6c9582952f93423e4ed8c6de452a28378de55f9238e1869fbe720f2e58292e WatchSource:0}: Error finding container dd6c9582952f93423e4ed8c6de452a28378de55f9238e1869fbe720f2e58292e: Status 404 returned error can't find the container with id dd6c9582952f93423e4ed8c6de452a28378de55f9238e1869fbe720f2e58292e Nov 02 17:47:21.241036 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:21.241006 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5"] Nov 02 17:47:21.277995 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:21.277920 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea104436-796f66c9bc-fdrgl" podUID="0891f7a9-f1df-49d5-b085-e2c343a05de5" containerName="payload-pre-processing" containerID="cri-o://89b533fdbe15f6ee9b9e0d51481887af5caa3b3911597be2b3f7d05adee47981" gracePeriod=25 Nov 02 17:47:21.322041 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:21.321990 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:47:21.322163 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:21.322097 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:47:21.322163 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:21.322147 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:47:21.918756 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:21.918719 4471 generic.go:356] "Generic (PLEG): container finished" podID="0891f7a9-f1df-49d5-b085-e2c343a05de5" containerID="89b533fdbe15f6ee9b9e0d51481887af5caa3b3911597be2b3f7d05adee47981" exitCode=0 Nov 02 17:47:21.919141 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:21.919096 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea104436-796f66c9bc-fdrgl" event={"ID":"0891f7a9-f1df-49d5-b085-e2c343a05de5","Type":"ContainerDied","Data":"89b533fdbe15f6ee9b9e0d51481887af5caa3b3911597be2b3f7d05adee47981"} Nov 02 17:47:21.984332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:21.976472 4471 generic.go:356] "Generic (PLEG): container finished" podID="b1b56137-0532-4dff-bfbe-9ae561d67562" containerID="def5f29ff75cd886d3b16754ef67f3532f70fc10eec6e84a78533188aa139924" exitCode=0 Nov 02 17:47:21.984797 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:21.984662 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ea104436-67cd4cf869-sk2pc" event={"ID":"b1b56137-0532-4dff-bfbe-9ae561d67562","Type":"ContainerDied","Data":"def5f29ff75cd886d3b16754ef67f3532f70fc10eec6e84a78533188aa139924"} Nov 02 17:47:22.122399 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:22.094655 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" event={"ID":"1a1b8c9e-6e63-4397-b3c0-46055331baa5","Type":"ContainerStarted","Data":"dfc08423a79920903da848413168593d223ea2ccdd2804196f45424fc1eacd18"} Nov 02 17:47:22.122399 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:22.094731 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" event={"ID":"1a1b8c9e-6e63-4397-b3c0-46055331baa5","Type":"ContainerStarted","Data":"dd6c9582952f93423e4ed8c6de452a28378de55f9238e1869fbe720f2e58292e"} Nov 02 17:47:22.387432 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:22.386317 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ea104436-67cd4cf869-sk2pc" Nov 02 17:47:22.596337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:22.585633 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" podStartSLOduration=3.585611249 podStartE2EDuration="3.585611249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:22.179934922 +0000 UTC m=+1059.875290424" watchObservedRunningTime="2026-11-02 17:47:22.585611249 +0000 UTC m=+1060.280966722" Nov 02 17:47:23.431377 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.419502 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1b56137-0532-4dff-bfbe-9ae561d67562-kube-api-access-r9k8w\" (UniqueName: \"kubernetes.io/projected/b1b56137-0532-4dff-bfbe-9ae561d67562-kube-api-access-r9k8w\") pod \"b1b56137-0532-4dff-bfbe-9ae561d67562\" (UID: \"b1b56137-0532-4dff-bfbe-9ae561d67562\") " Nov 02 17:47:23.431377 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.419894 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1b56137-0532-4dff-bfbe-9ae561d67562-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1b56137-0532-4dff-bfbe-9ae561d67562-plugins-config-volume\") pod \"b1b56137-0532-4dff-bfbe-9ae561d67562\" (UID: \"b1b56137-0532-4dff-bfbe-9ae561d67562\") " Nov 02 17:47:23.446401 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.433920 4471 patch_prober.go:29] interesting pod/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:47:23.446401 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.434001 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" podUID="7ebdad0f-3d23-452f-83dc-55fed8615cf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:47:23.454387 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.454292 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 02 17:47:23.471684 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.471638 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1b56137-0532-4dff-bfbe-9ae561d67562-plugins-config-volume" pod "b1b56137-0532-4dff-bfbe-9ae561d67562" (UID: "b1b56137-0532-4dff-bfbe-9ae561d67562"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:47:23.529349 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.523783 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b56137-0532-4dff-bfbe-9ae561d67562-kube-api-access-r9k8w" pod "b1b56137-0532-4dff-bfbe-9ae561d67562" (UID: "b1b56137-0532-4dff-bfbe-9ae561d67562"). InnerVolumeSpecName "kube-api-access-r9k8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:47:23.534669 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.534625 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:47:23.551130 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.551095 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1b56137-0532-4dff-bfbe-9ae561d67562-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:23.551383 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.551353 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9k8w\" (UniqueName: \"kubernetes.io/projected/b1b56137-0532-4dff-bfbe-9ae561d67562-kube-api-access-r9k8w\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:23.677333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.585440 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:47:23.677333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.585542 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 02 17:47:23.677333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.585626 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:47:23.677333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.600221 4471 patch_prober.go:29] interesting pod/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 02 17:47:23.677333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.600275 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" podUID="1a1b8c9e-6e63-4397-b3c0-46055331baa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 02 17:47:23.677333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.648142 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea104436-796f66c9bc-fdrgl" Nov 02 17:47:23.703386 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.696423 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:47:23.703386 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:47:23.699036 4471 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.235s" Nov 02 17:47:23.716944 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.716905 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0891f7a9-f1df-49d5-b085-e2c343a05de5-kube-api-access-8jljb\" (UniqueName: \"kubernetes.io/projected/0891f7a9-f1df-49d5-b085-e2c343a05de5-kube-api-access-8jljb\") pod \"0891f7a9-f1df-49d5-b085-e2c343a05de5\" (UID: \"0891f7a9-f1df-49d5-b085-e2c343a05de5\") " Nov 02 17:47:23.763330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.752609 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0891f7a9-f1df-49d5-b085-e2c343a05de5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0891f7a9-f1df-49d5-b085-e2c343a05de5-plugins-config-volume\") pod \"0891f7a9-f1df-49d5-b085-e2c343a05de5\" (UID: \"0891f7a9-f1df-49d5-b085-e2c343a05de5\") " Nov 02 17:47:23.763330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.758705 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0891f7a9-f1df-49d5-b085-e2c343a05de5-plugins-config-volume" pod "0891f7a9-f1df-49d5-b085-e2c343a05de5" (UID: "0891f7a9-f1df-49d5-b085-e2c343a05de5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:47:23.763330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.744932 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" Nov 02 17:47:23.763330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.758896 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea104436-796f66c9bc-fdrgl" event={"ID":"0891f7a9-f1df-49d5-b085-e2c343a05de5","Type":"ContainerDied","Data":"e9d4ac9c67908ef8521c6e6ebadc5ad32053f70e5f72b6b8ab0aebf720eab6a4"} Nov 02 17:47:23.763330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.758945 4471 scope.go:122] "RemoveContainer" containerID="89b533fdbe15f6ee9b9e0d51481887af5caa3b3911597be2b3f7d05adee47981" Nov 02 17:47:23.798357 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.794968 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0891f7a9-f1df-49d5-b085-e2c343a05de5-kube-api-access-8jljb" pod "0891f7a9-f1df-49d5-b085-e2c343a05de5" (UID: "0891f7a9-f1df-49d5-b085-e2c343a05de5"). InnerVolumeSpecName "kube-api-access-8jljb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:47:23.817367 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.811066 4471 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" secret="" err="secret \"e2e-derive-93321a4f-openshift-default-dockercfg-8fp78\" not found" Nov 02 17:47:23.817367 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.811886 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ea104436-67cd4cf869-sk2pc" Nov 02 17:47:23.860352 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.823664 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ea104436-67cd4cf869-sk2pc" event={"ID":"b1b56137-0532-4dff-bfbe-9ae561d67562","Type":"ContainerDied","Data":"ab2c0300e7213a0b1dfeea8d9a844378624a7f0274d86e28ee3f1ef225856c93"} Nov 02 17:47:23.905526 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.880017 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd"] Nov 02 17:47:23.905526 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.881179 4471 cpu_manager.go:414] "containerMap: removing container" podUID="0891f7a9-f1df-49d5-b085-e2c343a05de5" containerName="payload-pre-processing" Nov 02 17:47:23.905526 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.881200 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0891f7a9-f1df-49d5-b085-e2c343a05de5" containerName="payload-pre-processing" Nov 02 17:47:23.905526 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.881221 4471 cpu_manager.go:414] "containerMap: removing container" podUID="b1b56137-0532-4dff-bfbe-9ae561d67562" containerName="payload-processing" Nov 02 17:47:23.905526 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.881230 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1b56137-0532-4dff-bfbe-9ae561d67562" containerName="payload-processing" Nov 02 17:47:23.905526 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.889629 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0891f7a9-f1df-49d5-b085-e2c343a05de5-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:23.905526 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.889666 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jljb\" (UniqueName: \"kubernetes.io/projected/0891f7a9-f1df-49d5-b085-e2c343a05de5-kube-api-access-8jljb\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:23.905526 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.896967 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-93321a4f-684bd94557-hlncd" podUID="62bbf303-13e9-4002-9236-253bd2d76062" containerName="payload-processing" containerID="cri-o://33064384c433ff89315d37528c2c7412b6e6eb89cd80fbb5d8e52f17df6d8777" gracePeriod=25 Nov 02 17:47:23.908635 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.906387 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="0891f7a9-f1df-49d5-b085-e2c343a05de5" containerName="payload-pre-processing" Nov 02 17:47:23.908635 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.906419 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1b56137-0532-4dff-bfbe-9ae561d67562" containerName="payload-processing" Nov 02 17:47:23.910934 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.910909 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd"] Nov 02 17:47:23.913581 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.913371 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx"] Nov 02 17:47:23.913581 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.913569 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" podUID="7ebdad0f-3d23-452f-83dc-55fed8615cf7" containerName="istio-proxy" containerID="cri-o://178dae968066919b5d15597b76945c747d76c3251422482689e97b560bb9db6d" gracePeriod=30 Nov 02 17:47:23.913761 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.913739 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:47:23.914294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.914272 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5"] Nov 02 17:47:23.927012 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.926791 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-93321a4f-6fd4889ff5-r5gg6" podUID="748d8213-fb65-48ff-8280-fd153833f578" containerName="payload-pre-processing" containerID="cri-o://efcfdda3160a0ca4db9eef37a139962bd88c451c222e57e99965cb1d9a289744" gracePeriod=25 Nov 02 17:47:23.932578 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.931740 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-f6da5f-openshift-default-dockercfg-wmv9z\"" Nov 02 17:47:24.005532 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.997874 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-istio-envoy\") pod \"e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:47:24.005532 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.997932 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-istio-data\") pod \"e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:47:24.005532 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.998015 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfc98\" (UniqueName: \"kubernetes.io/projected/3de46b2c-81e7-458e-b693-088684a8a8d8-kube-api-access-rfc98\") pod \"e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:47:24.005532 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.998086 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-credential-socket\") pod \"e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:47:24.005532 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.998134 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de46b2c-81e7-458e-b693-088684a8a8d8-istio-podinfo\") pod \"e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:47:24.005532 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.998225 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-workload-certs\") pod \"e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:47:24.005532 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.998334 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3de46b2c-81e7-458e-b693-088684a8a8d8-istio-token\") pod \"e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:47:24.005532 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.998428 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de46b2c-81e7-458e-b693-088684a8a8d8-istiod-ca-cert\") pod \"e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:47:24.005532 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:23.998470 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-workload-socket\") pod \"e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:47:24.078328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:24.069450 4471 patch_prober.go:29] interesting pod/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 02 17:47:24.078328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:24.069516 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" podUID="1a1b8c9e-6e63-4397-b3c0-46055331baa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 02 17:47:24.112883 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:24.103593 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3de46b2c-81e7-458e-b693-088684a8a8d8-istio-token\") pod \"e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:47:24.112883 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:24.103718 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de46b2c-81e7-458e-b693-088684a8a8d8-istiod-ca-cert\") pod \"e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:47:24.112883 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:24.103769 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-workload-socket\") pod \"e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:47:24.112883 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:24.103820 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-istio-envoy\") pod \"e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:47:24.112883 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:24.103862 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-istio-data\") pod \"e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:47:24.112883 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:24.104093 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfc98\" (UniqueName: \"kubernetes.io/projected/3de46b2c-81e7-458e-b693-088684a8a8d8-kube-api-access-rfc98\") pod \"e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:47:24.112883 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:24.104176 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-credential-socket\") pod \"e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:47:24.112883 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:24.104235 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de46b2c-81e7-458e-b693-088684a8a8d8-istio-podinfo\") pod \"e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:47:24.112883 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:24.104834 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-workload-socket\") pod \"e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:47:24.117411 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:24.113796 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-workload-certs\") pod \"e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:47:24.117411 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:24.114267 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-workload-certs\") pod \"e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:47:24.117411 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:24.114971 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-credential-socket\") pod \"e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:47:24.117411 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:24.116150 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de46b2c-81e7-458e-b693-088684a8a8d8-istiod-ca-cert\") pod \"e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:47:24.117411 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:24.116734 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-istio-data\") pod \"e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:47:24.156353 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:24.149929 4471 scope.go:122] "RemoveContainer" containerID="def5f29ff75cd886d3b16754ef67f3532f70fc10eec6e84a78533188aa139924" Nov 02 17:47:24.228021 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:24.219713 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ea104436-67cd4cf869-sk2pc"] Nov 02 17:47:24.285807 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:24.285777 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ea104436-67cd4cf869-sk2pc"] Nov 02 17:47:24.382675 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:47:24.382633 4471 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b56137_0532_4dff_bfbe_9ae561d67562.slice/crio-ab2c0300e7213a0b1dfeea8d9a844378624a7f0274d86e28ee3f1ef225856c93\": RecentStats: unable to find data in memory cache]" Nov 02 17:47:24.405435 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:24.397999 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfc98\" (UniqueName: \"kubernetes.io/projected/3de46b2c-81e7-458e-b693-088684a8a8d8-kube-api-access-rfc98\") pod \"e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:47:24.552636 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:24.552601 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3de46b2c-81e7-458e-b693-088684a8a8d8-istio-token\") pod \"e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:47:24.553436 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:24.553411 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-istio-envoy\") pod \"e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:47:24.554073 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:24.554050 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de46b2c-81e7-458e-b693-088684a8a8d8-istio-podinfo\") pod \"e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:47:24.621017 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:24.620685 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:47:25.096969 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.089960 4471 patch_prober.go:29] interesting pod/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 02 17:47:25.096969 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.090025 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" podUID="1a1b8c9e-6e63-4397-b3c0-46055331baa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 02 17:47:25.117897 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.117646 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1b56137-0532-4dff-bfbe-9ae561d67562" path="/var/lib/kubelet/pods/b1b56137-0532-4dff-bfbe-9ae561d67562/volumes" Nov 02 17:47:25.123066 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.123032 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658"] Nov 02 17:47:25.162686 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.162652 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658"] Nov 02 17:47:25.163119 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.163096 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:47:25.185472 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.184298 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-f91b2d-openshift-default-dockercfg-st4mz\"" Nov 02 17:47:25.198189 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.197765 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea104436-796f66c9bc-fdrgl" Nov 02 17:47:25.198189 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.197770 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istio-data\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:47:25.198189 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.197965 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbs8z\" (UniqueName: \"kubernetes.io/projected/684a3084-e591-4e7c-8e0d-7f2d65211a2a-kube-api-access-dbs8z\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:47:25.198189 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.198063 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-workload-certs\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:47:25.200117 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.199793 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istio-podinfo\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:47:25.200117 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.199907 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-workload-socket\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:47:25.200117 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.200069 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istio-token\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:47:25.200459 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.200130 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-credential-socket\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:47:25.200459 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.200268 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istio-envoy\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:47:25.200459 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.200389 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istiod-ca-cert\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:47:25.241667 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.241595 4471 generic.go:356] "Generic (PLEG): container finished" podID="748d8213-fb65-48ff-8280-fd153833f578" containerID="efcfdda3160a0ca4db9eef37a139962bd88c451c222e57e99965cb1d9a289744" exitCode=0 Nov 02 17:47:25.241981 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.241645 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-93321a4f-6fd4889ff5-r5gg6" event={"ID":"748d8213-fb65-48ff-8280-fd153833f578","Type":"ContainerDied","Data":"efcfdda3160a0ca4db9eef37a139962bd88c451c222e57e99965cb1d9a289744"} Nov 02 17:47:25.246009 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.245979 4471 generic.go:356] "Generic (PLEG): container finished" podID="62bbf303-13e9-4002-9236-253bd2d76062" containerID="33064384c433ff89315d37528c2c7412b6e6eb89cd80fbb5d8e52f17df6d8777" exitCode=0 Nov 02 17:47:25.246694 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.246668 4471 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" secret="" err="secret \"e2e-derive-93321a4f-openshift-default-dockercfg-8fp78\" not found" Nov 02 17:47:25.247721 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.247694 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-93321a4f-684bd94557-hlncd" event={"ID":"62bbf303-13e9-4002-9236-253bd2d76062","Type":"ContainerDied","Data":"33064384c433ff89315d37528c2c7412b6e6eb89cd80fbb5d8e52f17df6d8777"} Nov 02 17:47:25.311557 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.309077 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbs8z\" (UniqueName: \"kubernetes.io/projected/684a3084-e591-4e7c-8e0d-7f2d65211a2a-kube-api-access-dbs8z\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:47:25.311557 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.309170 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-workload-certs\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:47:25.311557 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.309207 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istio-podinfo\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:47:25.311557 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.309257 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-workload-socket\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:47:25.311557 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.309395 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istio-token\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:47:25.311557 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.309470 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-credential-socket\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:47:25.311557 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.309611 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istio-envoy\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:47:25.311557 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.309681 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istiod-ca-cert\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:47:25.311557 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.309799 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istio-data\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:47:25.311557 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.310224 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istio-data\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:47:25.311557 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.310758 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-workload-certs\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:47:25.314544 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.314168 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-credential-socket\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:47:25.315545 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.314694 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-workload-socket\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:47:25.315545 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.315485 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istiod-ca-cert\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:47:25.328329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.324030 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istio-podinfo\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:47:25.335526 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.335459 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istio-envoy\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:47:25.346299 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.346262 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ea104436-796f66c9bc-fdrgl"] Nov 02 17:47:25.358448 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.358375 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ea104436-796f66c9bc-fdrgl"] Nov 02 17:47:25.362200 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.362122 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istio-token\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:47:25.526905 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.526854 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbs8z\" (UniqueName: \"kubernetes.io/projected/684a3084-e591-4e7c-8e0d-7f2d65211a2a-kube-api-access-dbs8z\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:47:25.631388 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.615398 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:47:25.760536 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.756693 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-93321a4f-684bd94557-hlncd" Nov 02 17:47:25.789107 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.789071 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-93321a4f-6fd4889ff5-r5gg6" Nov 02 17:47:25.866467 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.856077 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62bbf303-13e9-4002-9236-253bd2d76062-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62bbf303-13e9-4002-9236-253bd2d76062-plugins-config-volume\") pod \"62bbf303-13e9-4002-9236-253bd2d76062\" (UID: \"62bbf303-13e9-4002-9236-253bd2d76062\") " Nov 02 17:47:25.866467 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.856330 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62bbf303-13e9-4002-9236-253bd2d76062-kube-api-access-bgrt9\" (UniqueName: \"kubernetes.io/projected/62bbf303-13e9-4002-9236-253bd2d76062-kube-api-access-bgrt9\") pod \"62bbf303-13e9-4002-9236-253bd2d76062\" (UID: \"62bbf303-13e9-4002-9236-253bd2d76062\") " Nov 02 17:47:25.866467 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.865734 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62bbf303-13e9-4002-9236-253bd2d76062-plugins-config-volume" pod "62bbf303-13e9-4002-9236-253bd2d76062" (UID: "62bbf303-13e9-4002-9236-253bd2d76062"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:47:25.867905 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.867869 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62bbf303-13e9-4002-9236-253bd2d76062-kube-api-access-bgrt9" pod "62bbf303-13e9-4002-9236-253bd2d76062" (UID: "62bbf303-13e9-4002-9236-253bd2d76062"). InnerVolumeSpecName "kube-api-access-bgrt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:47:25.869481 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.869427 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgrt9\" (UniqueName: \"kubernetes.io/projected/62bbf303-13e9-4002-9236-253bd2d76062-kube-api-access-bgrt9\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:25.869481 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:25.869457 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62bbf303-13e9-4002-9236-253bd2d76062-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:26.026810 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:26.019936 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/748d8213-fb65-48ff-8280-fd153833f578-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/748d8213-fb65-48ff-8280-fd153833f578-plugins-config-volume\") pod \"748d8213-fb65-48ff-8280-fd153833f578\" (UID: \"748d8213-fb65-48ff-8280-fd153833f578\") " Nov 02 17:47:26.026810 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:26.020007 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/748d8213-fb65-48ff-8280-fd153833f578-kube-api-access-vftg7\" (UniqueName: \"kubernetes.io/projected/748d8213-fb65-48ff-8280-fd153833f578-kube-api-access-vftg7\") pod \"748d8213-fb65-48ff-8280-fd153833f578\" (UID: \"748d8213-fb65-48ff-8280-fd153833f578\") " Nov 02 17:47:26.026810 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:26.023058 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/748d8213-fb65-48ff-8280-fd153833f578-plugins-config-volume" pod "748d8213-fb65-48ff-8280-fd153833f578" (UID: "748d8213-fb65-48ff-8280-fd153833f578"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:47:26.028496 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:26.027497 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/748d8213-fb65-48ff-8280-fd153833f578-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:26.031027 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:26.029431 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/748d8213-fb65-48ff-8280-fd153833f578-kube-api-access-vftg7" pod "748d8213-fb65-48ff-8280-fd153833f578" (UID: "748d8213-fb65-48ff-8280-fd153833f578"). InnerVolumeSpecName "kube-api-access-vftg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:47:26.042350 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:26.042297 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd"] Nov 02 17:47:26.074954 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:26.074545 4471 patch_prober.go:29] interesting pod/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 02 17:47:26.074954 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:26.074596 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" podUID="1a1b8c9e-6e63-4397-b3c0-46055331baa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 02 17:47:26.077738 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:26.077682 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:47:26.077862 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:26.077804 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:47:26.077862 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:26.077850 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:47:26.121881 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:26.121841 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658"] Nov 02 17:47:26.122099 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:47:26.122066 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod684a3084_e591_4e7c_8e0d_7f2d65211a2a.slice/crio-37d60d1f59812c750bdc33682cb62601d6b92eaae4ad538829f0ffb75e20f24e WatchSource:0}: Error finding container 37d60d1f59812c750bdc33682cb62601d6b92eaae4ad538829f0ffb75e20f24e: Status 404 returned error can't find the container with id 37d60d1f59812c750bdc33682cb62601d6b92eaae4ad538829f0ffb75e20f24e Nov 02 17:47:26.125507 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:26.125474 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:47:26.125603 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:26.125544 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:47:26.125603 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:26.125580 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:47:26.129712 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:26.129673 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vftg7\" (UniqueName: \"kubernetes.io/projected/748d8213-fb65-48ff-8280-fd153833f578-kube-api-access-vftg7\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:26.257719 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:26.257665 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" event={"ID":"684a3084-e591-4e7c-8e0d-7f2d65211a2a","Type":"ContainerStarted","Data":"37d60d1f59812c750bdc33682cb62601d6b92eaae4ad538829f0ffb75e20f24e"} Nov 02 17:47:26.264770 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:26.259419 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-93321a4f-6fd4889ff5-r5gg6" Nov 02 17:47:26.264770 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:26.260840 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-93321a4f-6fd4889ff5-r5gg6" event={"ID":"748d8213-fb65-48ff-8280-fd153833f578","Type":"ContainerDied","Data":"999134bfee2f150b0b3b08374ee37a2f3254aff67febae834beb33792a4eb58a"} Nov 02 17:47:26.264770 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:26.260911 4471 scope.go:122] "RemoveContainer" containerID="efcfdda3160a0ca4db9eef37a139962bd88c451c222e57e99965cb1d9a289744" Nov 02 17:47:26.267676 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:26.267653 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" event={"ID":"3de46b2c-81e7-458e-b693-088684a8a8d8","Type":"ContainerStarted","Data":"8399a59c4ce7fc6b3a252d81d472281b29c65b0724f4fe40d60c4ee606590140"} Nov 02 17:47:26.267780 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:26.267695 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" event={"ID":"3de46b2c-81e7-458e-b693-088684a8a8d8","Type":"ContainerStarted","Data":"ca4e75cfd89534938f21011cf065b7abe4b4b8ac071a798887fe06d133913beb"} Nov 02 17:47:26.270324 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:26.270026 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" podUID="1a1b8c9e-6e63-4397-b3c0-46055331baa5" containerName="istio-proxy" containerID="cri-o://dfc08423a79920903da848413168593d223ea2ccdd2804196f45424fc1eacd18" gracePeriod=30 Nov 02 17:47:26.270324 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:26.270144 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-93321a4f-684bd94557-hlncd" event={"ID":"62bbf303-13e9-4002-9236-253bd2d76062","Type":"ContainerDied","Data":"57bdcfdf31eea48bc0dec778c41fd62ab7a7b3f3cd2a92993d0ab051057de1dc"} Nov 02 17:47:26.277469 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:26.277411 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-93321a4f-684bd94557-hlncd" Nov 02 17:47:26.294199 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:26.293824 4471 scope.go:122] "RemoveContainer" containerID="33064384c433ff89315d37528c2c7412b6e6eb89cd80fbb5d8e52f17df6d8777" Nov 02 17:47:26.421360 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:26.419905 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" podStartSLOduration=4.419883662 podStartE2EDuration="4.419883662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:26.416284787 +0000 UTC m=+1064.111640281" watchObservedRunningTime="2026-11-02 17:47:26.419883662 +0000 UTC m=+1064.115239139" Nov 02 17:47:26.462747 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:26.462699 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0891f7a9-f1df-49d5-b085-e2c343a05de5" path="/var/lib/kubelet/pods/0891f7a9-f1df-49d5-b085-e2c343a05de5/volumes" Nov 02 17:47:26.464956 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:26.464914 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-93321a4f-6fd4889ff5-r5gg6"] Nov 02 17:47:26.477755 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:26.477719 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-93321a4f-6fd4889ff5-r5gg6"] Nov 02 17:47:26.527364 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:26.524283 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-93321a4f-684bd94557-hlncd"] Nov 02 17:47:26.600398 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:26.594131 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-93321a4f-684bd94557-hlncd"] Nov 02 17:47:26.630767 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:26.630684 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:47:27.282841 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:27.282797 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" event={"ID":"684a3084-e591-4e7c-8e0d-7f2d65211a2a","Type":"ContainerStarted","Data":"7a32f19a3d93490aca8b7b3d76882c2fbc5b96cce7557f6cd9b19f62be53f705"} Nov 02 17:47:27.354470 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:27.354420 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" podStartSLOduration=3.354405957 podStartE2EDuration="3.354405957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:27.350566844 +0000 UTC m=+1065.045922318" watchObservedRunningTime="2026-11-02 17:47:27.354405957 +0000 UTC m=+1065.049761430" Nov 02 17:47:27.658400 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:27.639081 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:47:27.658400 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:27.650802 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:47:27.664966 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:27.663473 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:47:28.300571 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:28.300392 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:47:28.300571 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:28.300484 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:47:28.302704 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:28.302207 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:47:28.302704 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:28.302648 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:47:28.460026 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:28.458447 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62bbf303-13e9-4002-9236-253bd2d76062" path="/var/lib/kubelet/pods/62bbf303-13e9-4002-9236-253bd2d76062/volumes" Nov 02 17:47:28.460026 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:28.459097 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="748d8213-fb65-48ff-8280-fd153833f578" path="/var/lib/kubelet/pods/748d8213-fb65-48ff-8280-fd153833f578/volumes" Nov 02 17:47:28.460026 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:28.459641 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-504de7-786496d969-w8f9z"] Nov 02 17:47:28.460949 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:28.460636 4471 cpu_manager.go:414] "containerMap: removing container" podUID="62bbf303-13e9-4002-9236-253bd2d76062" containerName="payload-processing" Nov 02 17:47:28.460949 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:28.460658 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62bbf303-13e9-4002-9236-253bd2d76062" containerName="payload-processing" Nov 02 17:47:28.460949 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:28.460693 4471 cpu_manager.go:414] "containerMap: removing container" podUID="748d8213-fb65-48ff-8280-fd153833f578" containerName="payload-pre-processing" Nov 02 17:47:28.460949 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:28.460701 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="748d8213-fb65-48ff-8280-fd153833f578" containerName="payload-pre-processing" Nov 02 17:47:28.460949 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:28.460881 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="62bbf303-13e9-4002-9236-253bd2d76062" containerName="payload-processing" Nov 02 17:47:28.460949 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:28.460899 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="748d8213-fb65-48ff-8280-fd153833f578" containerName="payload-pre-processing" Nov 02 17:47:28.506435 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:28.506389 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-504de7-786496d969-w8f9z"] Nov 02 17:47:28.506630 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:28.506526 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-504de7-786496d969-w8f9z" Nov 02 17:47:28.585998 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:28.584765 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-504de7-dockercfg-d4zk4\"" Nov 02 17:47:28.596721 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:28.592590 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-504de7\"" Nov 02 17:47:28.648500 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:28.640444 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a06f649d-baff-4ee4-b35b-bc5966df3067-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-504de7-786496d969-w8f9z\" (UID: \"a06f649d-baff-4ee4-b35b-bc5966df3067\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-504de7-786496d969-w8f9z" Nov 02 17:47:28.648500 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:28.640664 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wn57\" (UniqueName: \"kubernetes.io/projected/a06f649d-baff-4ee4-b35b-bc5966df3067-kube-api-access-4wn57\") pod \"payload-processing-e2e-shared-a-w5-504de7-786496d969-w8f9z\" (UID: \"a06f649d-baff-4ee4-b35b-bc5966df3067\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-504de7-786496d969-w8f9z" Nov 02 17:47:28.762328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:28.756065 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wn57\" (UniqueName: \"kubernetes.io/projected/a06f649d-baff-4ee4-b35b-bc5966df3067-kube-api-access-4wn57\") pod \"payload-processing-e2e-shared-a-w5-504de7-786496d969-w8f9z\" (UID: \"a06f649d-baff-4ee4-b35b-bc5966df3067\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-504de7-786496d969-w8f9z" Nov 02 17:47:28.762328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:28.756552 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a06f649d-baff-4ee4-b35b-bc5966df3067-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-504de7-786496d969-w8f9z\" (UID: \"a06f649d-baff-4ee4-b35b-bc5966df3067\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-504de7-786496d969-w8f9z" Nov 02 17:47:28.762328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:28.759402 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a06f649d-baff-4ee4-b35b-bc5966df3067-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-504de7-786496d969-w8f9z\" (UID: \"a06f649d-baff-4ee4-b35b-bc5966df3067\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-504de7-786496d969-w8f9z" Nov 02 17:47:28.826344 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:28.819911 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wn57\" (UniqueName: \"kubernetes.io/projected/a06f649d-baff-4ee4-b35b-bc5966df3067-kube-api-access-4wn57\") pod \"payload-processing-e2e-shared-a-w5-504de7-786496d969-w8f9z\" (UID: \"a06f649d-baff-4ee4-b35b-bc5966df3067\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-504de7-786496d969-w8f9z" Nov 02 17:47:28.879411 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:28.877446 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-504de7-786496d969-w8f9z" Nov 02 17:47:30.362779 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:47:29.342080 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda06f649d_baff_4ee4_b35b_bc5966df3067.slice/crio-921673d410156d00ae64cd04a313b2ca8b514f3c38c33c999af90778604cf5d5 WatchSource:0}: Error finding container 921673d410156d00ae64cd04a313b2ca8b514f3c38c33c999af90778604cf5d5: Status 404 returned error can't find the container with id 921673d410156d00ae64cd04a313b2ca8b514f3c38c33c999af90778604cf5d5 Nov 02 17:47:30.362779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:29.344703 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-504de7-786496d969-w8f9z"] Nov 02 17:47:30.362779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:29.555028 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:47:30.362779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:29.578930 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istio-envoy\") pod \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " Nov 02 17:47:30.362779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:29.579016 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ebdad0f-3d23-452f-83dc-55fed8615cf7-kube-api-access-f69c6\" (UniqueName: \"kubernetes.io/projected/7ebdad0f-3d23-452f-83dc-55fed8615cf7-kube-api-access-f69c6\") pod \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " Nov 02 17:47:30.362779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:29.579048 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-workload-certs\") pod \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " Nov 02 17:47:30.362779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:29.579097 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istiod-ca-cert\") pod \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " Nov 02 17:47:30.362779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:29.579126 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-workload-socket\") pod \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " Nov 02 17:47:30.362779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:29.579145 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-credential-socket\") pod \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " Nov 02 17:47:30.362779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:29.579175 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istio-podinfo\") pod \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " Nov 02 17:47:30.362779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:29.579240 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istio-data\") pod \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " Nov 02 17:47:30.362779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:29.579354 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istio-token\" (UniqueName: \"kubernetes.io/projected/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istio-token\") pod \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\" (UID: \"7ebdad0f-3d23-452f-83dc-55fed8615cf7\") " Nov 02 17:47:30.362779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:29.579457 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-workload-certs" pod "7ebdad0f-3d23-452f-83dc-55fed8615cf7" (UID: "7ebdad0f-3d23-452f-83dc-55fed8615cf7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:47:30.362779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:29.579480 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-workload-socket" pod "7ebdad0f-3d23-452f-83dc-55fed8615cf7" (UID: "7ebdad0f-3d23-452f-83dc-55fed8615cf7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:47:30.362779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:29.579577 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-credential-socket" pod "7ebdad0f-3d23-452f-83dc-55fed8615cf7" (UID: "7ebdad0f-3d23-452f-83dc-55fed8615cf7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:47:30.362779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:29.579610 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istiod-ca-cert" pod "7ebdad0f-3d23-452f-83dc-55fed8615cf7" (UID: "7ebdad0f-3d23-452f-83dc-55fed8615cf7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:47:30.362779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:29.579730 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istio-data" pod "7ebdad0f-3d23-452f-83dc-55fed8615cf7" (UID: "7ebdad0f-3d23-452f-83dc-55fed8615cf7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:47:30.362779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:29.580252 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:30.362779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:29.580271 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:30.362779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:29.580280 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:30.362779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:29.580288 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:30.362779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:29.580296 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:30.362779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:29.582063 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istio-token" pod "7ebdad0f-3d23-452f-83dc-55fed8615cf7" (UID: "7ebdad0f-3d23-452f-83dc-55fed8615cf7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:47:30.362779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:29.582092 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ebdad0f-3d23-452f-83dc-55fed8615cf7-kube-api-access-f69c6" pod "7ebdad0f-3d23-452f-83dc-55fed8615cf7" (UID: "7ebdad0f-3d23-452f-83dc-55fed8615cf7"). InnerVolumeSpecName "kube-api-access-f69c6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:47:30.362779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:29.582256 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istio-podinfo" pod "7ebdad0f-3d23-452f-83dc-55fed8615cf7" (UID: "7ebdad0f-3d23-452f-83dc-55fed8615cf7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:47:30.362779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:29.582714 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istio-envoy" pod "7ebdad0f-3d23-452f-83dc-55fed8615cf7" (UID: "7ebdad0f-3d23-452f-83dc-55fed8615cf7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:47:30.362779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:29.681969 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:30.362779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:29.681997 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f69c6\" (UniqueName: \"kubernetes.io/projected/7ebdad0f-3d23-452f-83dc-55fed8615cf7-kube-api-access-f69c6\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:30.362779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:29.682006 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:30.362779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:29.682015 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ebdad0f-3d23-452f-83dc-55fed8615cf7-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:30.479874 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:30.473535 4471 generic.go:356] "Generic (PLEG): container finished" podID="7ebdad0f-3d23-452f-83dc-55fed8615cf7" containerID="178dae968066919b5d15597b76945c747d76c3251422482689e97b560bb9db6d" exitCode=0 Nov 02 17:47:30.479874 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:30.473880 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" Nov 02 17:47:30.563385 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:30.557056 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" event={"ID":"7ebdad0f-3d23-452f-83dc-55fed8615cf7","Type":"ContainerDied","Data":"178dae968066919b5d15597b76945c747d76c3251422482689e97b560bb9db6d"} Nov 02 17:47:30.563385 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:30.557136 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx" event={"ID":"7ebdad0f-3d23-452f-83dc-55fed8615cf7","Type":"ContainerDied","Data":"edbb12364b3f0790b115b7f02ee2e23f6bae4feae74585caa8328d111aada85f"} Nov 02 17:47:30.563385 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:30.558628 4471 scope.go:122] "RemoveContainer" containerID="178dae968066919b5d15597b76945c747d76c3251422482689e97b560bb9db6d" Nov 02 17:47:30.578402 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:30.577053 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-504de7-786496d969-w8f9z" event={"ID":"a06f649d-baff-4ee4-b35b-bc5966df3067","Type":"ContainerStarted","Data":"fe64d53bff10de18fe8ab223ec6ea4abacf9c938f07a88232c018222ce6746ff"} Nov 02 17:47:30.578402 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:30.577122 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-504de7-786496d969-w8f9z" event={"ID":"a06f649d-baff-4ee4-b35b-bc5966df3067","Type":"ContainerStarted","Data":"921673d410156d00ae64cd04a313b2ca8b514f3c38c33c999af90778604cf5d5"} Nov 02 17:47:30.578402 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:30.577699 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-504de7-786496d969-w8f9z" Nov 02 17:47:30.605787 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:30.605752 4471 scope.go:122] "RemoveContainer" containerID="178dae968066919b5d15597b76945c747d76c3251422482689e97b560bb9db6d" Nov 02 17:47:30.606359 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:47:30.606318 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"178dae968066919b5d15597b76945c747d76c3251422482689e97b560bb9db6d\": container with ID starting with 178dae968066919b5d15597b76945c747d76c3251422482689e97b560bb9db6d not found: ID does not exist" containerID="178dae968066919b5d15597b76945c747d76c3251422482689e97b560bb9db6d" Nov 02 17:47:30.606550 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:30.606374 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"178dae968066919b5d15597b76945c747d76c3251422482689e97b560bb9db6d"} err="failed to get container status \"178dae968066919b5d15597b76945c747d76c3251422482689e97b560bb9db6d\": rpc error: code = NotFound desc = could not find container \"178dae968066919b5d15597b76945c747d76c3251422482689e97b560bb9db6d\": container with ID starting with 178dae968066919b5d15597b76945c747d76c3251422482689e97b560bb9db6d not found: ID does not exist" Nov 02 17:47:30.653116 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:30.652971 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-504de7-786496d969-w8f9z" podStartSLOduration=2.652945014 podStartE2EDuration="2.652945014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:30.630752859 +0000 UTC m=+1068.326108359" watchObservedRunningTime="2026-11-02 17:47:30.652945014 +0000 UTC m=+1068.348300544" Nov 02 17:47:30.683601 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:30.683568 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx"] Nov 02 17:47:30.691467 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:30.691428 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-t84gx"] Nov 02 17:47:31.391861 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.391827 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx"] Nov 02 17:47:31.393229 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.393209 4471 cpu_manager.go:414] "containerMap: removing container" podUID="7ebdad0f-3d23-452f-83dc-55fed8615cf7" containerName="istio-proxy" Nov 02 17:47:31.393276 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.393232 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ebdad0f-3d23-452f-83dc-55fed8615cf7" containerName="istio-proxy" Nov 02 17:47:31.393381 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.393371 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ebdad0f-3d23-452f-83dc-55fed8615cf7" containerName="istio-proxy" Nov 02 17:47:31.401809 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.401787 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 17:47:31.411205 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.411179 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f6da5f-6cb75c9495-682jg"] Nov 02 17:47:31.419394 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.419370 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f6da5f-6cb75c9495-682jg" Nov 02 17:47:31.441024 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.440997 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-f6da5f\"" Nov 02 17:47:31.453686 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.452351 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx"] Nov 02 17:47:31.461952 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.461183 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istio-podinfo\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 17:47:31.461952 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.461275 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istiod-ca-cert\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 17:47:31.461952 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.461351 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-workload-socket\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 17:47:31.461952 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.461371 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istio-data\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 17:47:31.461952 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.461443 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-credential-socket\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 17:47:31.461952 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.461474 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istio-token\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 17:47:31.461952 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.461529 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kqx9\" (UniqueName: \"kubernetes.io/projected/d2d5291d-d4e9-462b-8f3e-f87f3b82d946-kube-api-access-7kqx9\") pod \"payload-pre-processing-e2e-shared-b-w6-f6da5f-6cb75c9495-682jg\" (UID: \"d2d5291d-d4e9-462b-8f3e-f87f3b82d946\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f6da5f-6cb75c9495-682jg" Nov 02 17:47:31.461952 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.461583 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istio-envoy\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 17:47:31.461952 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.461629 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d5291d-d4e9-462b-8f3e-f87f3b82d946-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-f6da5f-6cb75c9495-682jg\" (UID: \"d2d5291d-d4e9-462b-8f3e-f87f3b82d946\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f6da5f-6cb75c9495-682jg" Nov 02 17:47:31.461952 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.461721 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgmdr\" (UniqueName: \"kubernetes.io/projected/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-kube-api-access-sgmdr\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 17:47:31.461952 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.461747 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-workload-certs\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 17:47:31.463553 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.463525 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f6da5f-6cb75c9495-682jg"] Nov 02 17:47:31.494171 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.494146 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-f6da5f-dockercfg-zzgtn\"" Nov 02 17:47:31.670180 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.607670 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-workload-socket\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 17:47:31.670180 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.607725 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istio-data\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 17:47:31.670180 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.607844 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-credential-socket\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 17:47:31.670180 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.607872 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istio-token\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 17:47:31.670180 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.607928 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kqx9\" (UniqueName: \"kubernetes.io/projected/d2d5291d-d4e9-462b-8f3e-f87f3b82d946-kube-api-access-7kqx9\") pod \"payload-pre-processing-e2e-shared-b-w6-f6da5f-6cb75c9495-682jg\" (UID: \"d2d5291d-d4e9-462b-8f3e-f87f3b82d946\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f6da5f-6cb75c9495-682jg" Nov 02 17:47:31.670180 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.607995 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istio-envoy\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 17:47:31.670180 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.608045 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d5291d-d4e9-462b-8f3e-f87f3b82d946-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-f6da5f-6cb75c9495-682jg\" (UID: \"d2d5291d-d4e9-462b-8f3e-f87f3b82d946\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f6da5f-6cb75c9495-682jg" Nov 02 17:47:31.670180 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.608172 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgmdr\" (UniqueName: \"kubernetes.io/projected/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-kube-api-access-sgmdr\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 17:47:31.670180 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.608360 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-workload-certs\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 17:47:31.670180 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.608444 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istio-podinfo\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 17:47:31.670180 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.608555 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istiod-ca-cert\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 17:47:31.670180 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.609458 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istiod-ca-cert\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 17:47:31.670180 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.609744 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-workload-socket\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 17:47:31.670180 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.609961 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istio-data\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 17:47:31.670180 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.610175 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-credential-socket\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 17:47:31.697703 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.674133 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d5291d-d4e9-462b-8f3e-f87f3b82d946-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-f6da5f-6cb75c9495-682jg\" (UID: \"d2d5291d-d4e9-462b-8f3e-f87f3b82d946\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f6da5f-6cb75c9495-682jg" Nov 02 17:47:31.697703 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.675713 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-workload-certs\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 17:47:31.722110 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.721584 4471 generic.go:356] "Generic (PLEG): container finished" podID="1a1b8c9e-6e63-4397-b3c0-46055331baa5" containerID="dfc08423a79920903da848413168593d223ea2ccdd2804196f45424fc1eacd18" exitCode=0 Nov 02 17:47:31.722110 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.721729 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" event={"ID":"1a1b8c9e-6e63-4397-b3c0-46055331baa5","Type":"ContainerDied","Data":"dfc08423a79920903da848413168593d223ea2ccdd2804196f45424fc1eacd18"} Nov 02 17:47:31.729221 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.729189 4471 generic.go:356] "Generic (PLEG): container finished" podID="1ff9e12a-8b9c-4556-ae7a-fa3011dd516c" containerID="ab033f6cbb7efeb30c7490fd232a38b8081b46fcd5aa84659a37c98af3341d44" exitCode=0 Nov 02 17:47:31.730516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.730472 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" event={"ID":"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c","Type":"ContainerDied","Data":"ab033f6cbb7efeb30c7490fd232a38b8081b46fcd5aa84659a37c98af3341d44"} Nov 02 17:47:31.731135 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.731116 4471 scope.go:122] "RemoveContainer" containerID="ab033f6cbb7efeb30c7490fd232a38b8081b46fcd5aa84659a37c98af3341d44" Nov 02 17:47:31.734080 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.734048 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:47:31.734272 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.734250 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:47:31.734448 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.734428 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:47:31.789449 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.789391 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-f6da5f-57dbdd6f67-xsfnw"] Nov 02 17:47:31.823551 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.823521 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-f6da5f-57dbdd6f67-xsfnw"] Nov 02 17:47:31.823934 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:31.823915 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f6da5f-57dbdd6f67-xsfnw" Nov 02 17:47:32.017432 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.017386 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istio-token\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 17:47:32.018125 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.018088 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kqx9\" (UniqueName: \"kubernetes.io/projected/d2d5291d-d4e9-462b-8f3e-f87f3b82d946-kube-api-access-7kqx9\") pod \"payload-pre-processing-e2e-shared-b-w6-f6da5f-6cb75c9495-682jg\" (UID: \"d2d5291d-d4e9-462b-8f3e-f87f3b82d946\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f6da5f-6cb75c9495-682jg" Nov 02 17:47:32.019993 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.019957 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgmdr\" (UniqueName: \"kubernetes.io/projected/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-kube-api-access-sgmdr\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 17:47:32.020276 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.020242 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istio-envoy\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 17:47:32.020777 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.020756 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istio-podinfo\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 17:47:32.047359 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.043025 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 17:47:32.047359 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.045246 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zshjm\" (UniqueName: \"kubernetes.io/projected/00044b6e-d520-4934-ba8a-b2d638f899d6-kube-api-access-zshjm\") pod \"payload-processing-e2e-shared-b-w6-f6da5f-57dbdd6f67-xsfnw\" (UID: \"00044b6e-d520-4934-ba8a-b2d638f899d6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f6da5f-57dbdd6f67-xsfnw" Nov 02 17:47:32.047359 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.045387 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00044b6e-d520-4934-ba8a-b2d638f899d6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-f6da5f-57dbdd6f67-xsfnw\" (UID: \"00044b6e-d520-4934-ba8a-b2d638f899d6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f6da5f-57dbdd6f67-xsfnw" Nov 02 17:47:32.080333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.052467 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f6da5f-6cb75c9495-682jg" Nov 02 17:47:32.099726 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.097602 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" Nov 02 17:47:32.147414 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.147368 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00044b6e-d520-4934-ba8a-b2d638f899d6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-f6da5f-57dbdd6f67-xsfnw\" (UID: \"00044b6e-d520-4934-ba8a-b2d638f899d6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f6da5f-57dbdd6f67-xsfnw" Nov 02 17:47:32.148660 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.148627 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zshjm\" (UniqueName: \"kubernetes.io/projected/00044b6e-d520-4934-ba8a-b2d638f899d6-kube-api-access-zshjm\") pod \"payload-processing-e2e-shared-b-w6-f6da5f-57dbdd6f67-xsfnw\" (UID: \"00044b6e-d520-4934-ba8a-b2d638f899d6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f6da5f-57dbdd6f67-xsfnw" Nov 02 17:47:32.149933 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.149836 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00044b6e-d520-4934-ba8a-b2d638f899d6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-f6da5f-57dbdd6f67-xsfnw\" (UID: \"00044b6e-d520-4934-ba8a-b2d638f899d6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f6da5f-57dbdd6f67-xsfnw" Nov 02 17:47:32.190374 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.182153 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zshjm\" (UniqueName: \"kubernetes.io/projected/00044b6e-d520-4934-ba8a-b2d638f899d6-kube-api-access-zshjm\") pod \"payload-processing-e2e-shared-b-w6-f6da5f-57dbdd6f67-xsfnw\" (UID: \"00044b6e-d520-4934-ba8a-b2d638f899d6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f6da5f-57dbdd6f67-xsfnw" Nov 02 17:47:32.253349 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.252532 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istio-podinfo\") pod \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " Nov 02 17:47:32.253349 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.252698 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-workload-socket\") pod \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " Nov 02 17:47:32.253349 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.252800 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-credential-socket\") pod \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " Nov 02 17:47:32.253349 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.252855 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-workload-certs\") pod \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " Nov 02 17:47:32.253349 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.252916 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a1b8c9e-6e63-4397-b3c0-46055331baa5-kube-api-access-kmpsr\" (UniqueName: \"kubernetes.io/projected/1a1b8c9e-6e63-4397-b3c0-46055331baa5-kube-api-access-kmpsr\") pod \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " Nov 02 17:47:32.253349 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.253002 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istio-data\") pod \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " Nov 02 17:47:32.253349 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.253073 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istio-envoy\") pod \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " Nov 02 17:47:32.253349 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.253155 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istio-token\" (UniqueName: \"kubernetes.io/projected/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istio-token\") pod \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " Nov 02 17:47:32.253349 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.253211 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istiod-ca-cert\") pod \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\" (UID: \"1a1b8c9e-6e63-4397-b3c0-46055331baa5\") " Nov 02 17:47:32.255393 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.255008 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istiod-ca-cert" pod "1a1b8c9e-6e63-4397-b3c0-46055331baa5" (UID: "1a1b8c9e-6e63-4397-b3c0-46055331baa5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:47:32.266930 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.266724 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istio-podinfo" pod "1a1b8c9e-6e63-4397-b3c0-46055331baa5" (UID: "1a1b8c9e-6e63-4397-b3c0-46055331baa5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:47:32.267271 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.267094 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-workload-socket" pod "1a1b8c9e-6e63-4397-b3c0-46055331baa5" (UID: "1a1b8c9e-6e63-4397-b3c0-46055331baa5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:47:32.268039 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.267871 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-credential-socket" pod "1a1b8c9e-6e63-4397-b3c0-46055331baa5" (UID: "1a1b8c9e-6e63-4397-b3c0-46055331baa5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:47:32.268426 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.268150 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-workload-certs" pod "1a1b8c9e-6e63-4397-b3c0-46055331baa5" (UID: "1a1b8c9e-6e63-4397-b3c0-46055331baa5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:47:32.272528 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.272474 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istio-data" pod "1a1b8c9e-6e63-4397-b3c0-46055331baa5" (UID: "1a1b8c9e-6e63-4397-b3c0-46055331baa5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:47:32.278432 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.278392 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istio-envoy" pod "1a1b8c9e-6e63-4397-b3c0-46055331baa5" (UID: "1a1b8c9e-6e63-4397-b3c0-46055331baa5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:47:32.278532 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.278476 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istio-token" pod "1a1b8c9e-6e63-4397-b3c0-46055331baa5" (UID: "1a1b8c9e-6e63-4397-b3c0-46055331baa5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:47:32.282952 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.281664 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a1b8c9e-6e63-4397-b3c0-46055331baa5-kube-api-access-kmpsr" pod "1a1b8c9e-6e63-4397-b3c0-46055331baa5" (UID: "1a1b8c9e-6e63-4397-b3c0-46055331baa5"). InnerVolumeSpecName "kube-api-access-kmpsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:47:32.356960 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.356925 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmpsr\" (UniqueName: \"kubernetes.io/projected/1a1b8c9e-6e63-4397-b3c0-46055331baa5-kube-api-access-kmpsr\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:32.357088 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.356965 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:32.357088 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.356982 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:32.357088 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.356996 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:32.357088 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.357010 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:32.357088 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.357023 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a1b8c9e-6e63-4397-b3c0-46055331baa5-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:32.357088 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.357037 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:32.357088 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.357050 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:32.357088 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.357062 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a1b8c9e-6e63-4397-b3c0-46055331baa5-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:32.386083 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.386056 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f6da5f-57dbdd6f67-xsfnw" Nov 02 17:47:32.461238 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.461211 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ebdad0f-3d23-452f-83dc-55fed8615cf7" path="/var/lib/kubelet/pods/7ebdad0f-3d23-452f-83dc-55fed8615cf7/volumes" Nov 02 17:47:32.491077 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.490917 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx"] Nov 02 17:47:32.491077 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:47:32.491044 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5bb3fbe_93ce_44aa_9cf1_02394cc28af7.slice/crio-706cdf45b1d340dc75ecbff3678f6258d9bbdd3173e2c274874d0e56855389b7 WatchSource:0}: Error finding container 706cdf45b1d340dc75ecbff3678f6258d9bbdd3173e2c274874d0e56855389b7: Status 404 returned error can't find the container with id 706cdf45b1d340dc75ecbff3678f6258d9bbdd3173e2c274874d0e56855389b7 Nov 02 17:47:32.494930 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.494896 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:47:32.495019 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.494971 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:47:32.495077 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.495018 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:47:32.495739 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:47:32.495717 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d5291d_d4e9_462b_8f3e_f87f3b82d946.slice/crio-e6c8ec8ed16010d2393c12ff9e56db44449777a52e5c53a02998136891e76cbb WatchSource:0}: Error finding container e6c8ec8ed16010d2393c12ff9e56db44449777a52e5c53a02998136891e76cbb: Status 404 returned error can't find the container with id e6c8ec8ed16010d2393c12ff9e56db44449777a52e5c53a02998136891e76cbb Nov 02 17:47:32.495809 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.495784 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:47:32.497146 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.497123 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f6da5f-6cb75c9495-682jg"] Nov 02 17:47:32.571568 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.569621 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-credential-socket\") pod \"613f1277-5dc5-422e-8b92-d9be8ac858c4\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " Nov 02 17:47:32.571568 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.569677 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-workload-certs\") pod \"613f1277-5dc5-422e-8b92-d9be8ac858c4\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " Nov 02 17:47:32.571568 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.569699 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/613f1277-5dc5-422e-8b92-d9be8ac858c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/613f1277-5dc5-422e-8b92-d9be8ac858c4-istiod-ca-cert\") pod \"613f1277-5dc5-422e-8b92-d9be8ac858c4\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " Nov 02 17:47:32.571568 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.569817 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/613f1277-5dc5-422e-8b92-d9be8ac858c4-istio-token\" (UniqueName: \"kubernetes.io/projected/613f1277-5dc5-422e-8b92-d9be8ac858c4-istio-token\") pod \"613f1277-5dc5-422e-8b92-d9be8ac858c4\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " Nov 02 17:47:32.571568 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.569919 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-workload-socket\") pod \"613f1277-5dc5-422e-8b92-d9be8ac858c4\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " Nov 02 17:47:32.571568 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.569941 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/613f1277-5dc5-422e-8b92-d9be8ac858c4-kube-api-access-n4jm7\" (UniqueName: \"kubernetes.io/projected/613f1277-5dc5-422e-8b92-d9be8ac858c4-kube-api-access-n4jm7\") pod \"613f1277-5dc5-422e-8b92-d9be8ac858c4\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " Nov 02 17:47:32.571568 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.569962 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-istio-envoy\") pod \"613f1277-5dc5-422e-8b92-d9be8ac858c4\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " Nov 02 17:47:32.571568 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.569999 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-istio-data\") pod \"613f1277-5dc5-422e-8b92-d9be8ac858c4\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " Nov 02 17:47:32.571568 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.570076 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/613f1277-5dc5-422e-8b92-d9be8ac858c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/613f1277-5dc5-422e-8b92-d9be8ac858c4-istio-podinfo\") pod \"613f1277-5dc5-422e-8b92-d9be8ac858c4\" (UID: \"613f1277-5dc5-422e-8b92-d9be8ac858c4\") " Nov 02 17:47:32.571568 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.570574 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-workload-socket" pod "613f1277-5dc5-422e-8b92-d9be8ac858c4" (UID: "613f1277-5dc5-422e-8b92-d9be8ac858c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:47:32.571568 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.571002 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:32.571568 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.571222 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-istio-data" pod "613f1277-5dc5-422e-8b92-d9be8ac858c4" (UID: "613f1277-5dc5-422e-8b92-d9be8ac858c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:47:32.575899 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.575557 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-istio-envoy" pod "613f1277-5dc5-422e-8b92-d9be8ac858c4" (UID: "613f1277-5dc5-422e-8b92-d9be8ac858c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:47:32.576354 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.576296 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-credential-socket" pod "613f1277-5dc5-422e-8b92-d9be8ac858c4" (UID: "613f1277-5dc5-422e-8b92-d9be8ac858c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:47:32.576508 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.576448 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/613f1277-5dc5-422e-8b92-d9be8ac858c4-istiod-ca-cert" pod "613f1277-5dc5-422e-8b92-d9be8ac858c4" (UID: "613f1277-5dc5-422e-8b92-d9be8ac858c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:47:32.589587 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.583505 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613f1277-5dc5-422e-8b92-d9be8ac858c4-kube-api-access-n4jm7" pod "613f1277-5dc5-422e-8b92-d9be8ac858c4" (UID: "613f1277-5dc5-422e-8b92-d9be8ac858c4"). InnerVolumeSpecName "kube-api-access-n4jm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:47:32.589587 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.584824 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-workload-certs" pod "613f1277-5dc5-422e-8b92-d9be8ac858c4" (UID: "613f1277-5dc5-422e-8b92-d9be8ac858c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:47:32.595189 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.595150 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/613f1277-5dc5-422e-8b92-d9be8ac858c4-istio-podinfo" pod "613f1277-5dc5-422e-8b92-d9be8ac858c4" (UID: "613f1277-5dc5-422e-8b92-d9be8ac858c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:47:32.611352 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.611284 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613f1277-5dc5-422e-8b92-d9be8ac858c4-istio-token" pod "613f1277-5dc5-422e-8b92-d9be8ac858c4" (UID: "613f1277-5dc5-422e-8b92-d9be8ac858c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:47:32.686875 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.686836 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/613f1277-5dc5-422e-8b92-d9be8ac858c4-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:32.686875 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.686870 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:32.686875 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.686884 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:32.687187 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.686900 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/613f1277-5dc5-422e-8b92-d9be8ac858c4-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:32.687187 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.686914 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/613f1277-5dc5-422e-8b92-d9be8ac858c4-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:32.687187 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.686928 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4jm7\" (UniqueName: \"kubernetes.io/projected/613f1277-5dc5-422e-8b92-d9be8ac858c4-kube-api-access-n4jm7\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:32.687187 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.686940 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:32.687187 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.686953 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/613f1277-5dc5-422e-8b92-d9be8ac858c4-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:47:32.742794 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.742764 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-f6da5f-57dbdd6f67-xsfnw"] Nov 02 17:47:32.750080 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.750050 4471 generic.go:356] "Generic (PLEG): container finished" podID="613f1277-5dc5-422e-8b92-d9be8ac858c4" containerID="053c1cc3d43c4bb595a4e790118314ee2260d473843fe0590ff297f6845112d7" exitCode=0 Nov 02 17:47:32.750210 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.750181 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" event={"ID":"613f1277-5dc5-422e-8b92-d9be8ac858c4","Type":"ContainerDied","Data":"053c1cc3d43c4bb595a4e790118314ee2260d473843fe0590ff297f6845112d7"} Nov 02 17:47:32.750271 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.750228 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" event={"ID":"613f1277-5dc5-422e-8b92-d9be8ac858c4","Type":"ContainerDied","Data":"9af1aa617392a9321de03772c236a80ccedc1a8052ca420a99e4e6f25544c89a"} Nov 02 17:47:32.750271 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.750256 4471 scope.go:122] "RemoveContainer" containerID="053c1cc3d43c4bb595a4e790118314ee2260d473843fe0590ff297f6845112d7" Nov 02 17:47:32.750498 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.750405 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt" Nov 02 17:47:32.759632 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.759349 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f6da5f-6cb75c9495-682jg" event={"ID":"d2d5291d-d4e9-462b-8f3e-f87f3b82d946","Type":"ContainerStarted","Data":"8ad5d40ad115398ef98da8a1639a5c44dde7465d108d0f715edb66c7188bfbe0"} Nov 02 17:47:32.759632 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.759427 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f6da5f-6cb75c9495-682jg" Nov 02 17:47:32.759632 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.759473 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f6da5f-6cb75c9495-682jg" event={"ID":"d2d5291d-d4e9-462b-8f3e-f87f3b82d946","Type":"ContainerStarted","Data":"e6c8ec8ed16010d2393c12ff9e56db44449777a52e5c53a02998136891e76cbb"} Nov 02 17:47:32.765034 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.764226 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" Nov 02 17:47:32.765034 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.764228 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5" event={"ID":"1a1b8c9e-6e63-4397-b3c0-46055331baa5","Type":"ContainerDied","Data":"dd6c9582952f93423e4ed8c6de452a28378de55f9238e1869fbe720f2e58292e"} Nov 02 17:47:32.774009 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.773820 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" event={"ID":"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7","Type":"ContainerStarted","Data":"f7e74a5eade9d1084418f6b7343fd52192b0e9e291fbf6b5293314890e6f55d4"} Nov 02 17:47:32.774009 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.773884 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" event={"ID":"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7","Type":"ContainerStarted","Data":"706cdf45b1d340dc75ecbff3678f6258d9bbdd3173e2c274874d0e56855389b7"} Nov 02 17:47:32.777290 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.777266 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" event={"ID":"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c","Type":"ContainerStarted","Data":"aee5c0fe15357f50fe65ff36dc095bf45753eb353079bbf5ddc997f8ac441398"} Nov 02 17:47:32.777857 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.777790 4471 scope.go:122] "RemoveContainer" containerID="053c1cc3d43c4bb595a4e790118314ee2260d473843fe0590ff297f6845112d7" Nov 02 17:47:32.778345 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:47:32.778210 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"053c1cc3d43c4bb595a4e790118314ee2260d473843fe0590ff297f6845112d7\": container with ID starting with 053c1cc3d43c4bb595a4e790118314ee2260d473843fe0590ff297f6845112d7 not found: ID does not exist" containerID="053c1cc3d43c4bb595a4e790118314ee2260d473843fe0590ff297f6845112d7" Nov 02 17:47:32.778345 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.778241 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"053c1cc3d43c4bb595a4e790118314ee2260d473843fe0590ff297f6845112d7"} err="failed to get container status \"053c1cc3d43c4bb595a4e790118314ee2260d473843fe0590ff297f6845112d7\": rpc error: code = NotFound desc = could not find container \"053c1cc3d43c4bb595a4e790118314ee2260d473843fe0590ff297f6845112d7\": container with ID starting with 053c1cc3d43c4bb595a4e790118314ee2260d473843fe0590ff297f6845112d7 not found: ID does not exist" Nov 02 17:47:32.778345 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.778262 4471 scope.go:122] "RemoveContainer" containerID="dfc08423a79920903da848413168593d223ea2ccdd2804196f45424fc1eacd18" Nov 02 17:47:32.820201 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.820145 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f6da5f-6cb75c9495-682jg" podStartSLOduration=1.820126915 podStartE2EDuration="1.820126915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:32.81681377 +0000 UTC m=+1070.512169261" watchObservedRunningTime="2026-11-02 17:47:32.820126915 +0000 UTC m=+1070.515482389" Nov 02 17:47:32.905031 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.904997 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt"] Nov 02 17:47:32.915447 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.915406 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-ea104436-gw-openshift-default-56b58c74b5-pkvbt"] Nov 02 17:47:32.956579 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.955722 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5"] Nov 02 17:47:32.963879 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:32.963808 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-z9ns5"] Nov 02 17:47:33.049594 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:33.049564 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 17:47:33.049973 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:33.049945 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 02 17:47:33.050155 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:33.050130 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" podUID="e5bb3fbe-93ce-44aa-9cf1-02394cc28af7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 02 17:47:33.082336 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:33.082197 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" podStartSLOduration=2.082173913 podStartE2EDuration="2.082173913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:33.002693143 +0000 UTC m=+1070.698048618" watchObservedRunningTime="2026-11-02 17:47:33.082173913 +0000 UTC m=+1070.777529387" Nov 02 17:47:33.614917 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:33.614825 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-f91b2d-55449f7d65-nqbd9"] Nov 02 17:47:33.631226 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:33.631176 4471 cpu_manager.go:414] "containerMap: removing container" podUID="613f1277-5dc5-422e-8b92-d9be8ac858c4" containerName="istio-proxy" Nov 02 17:47:33.631474 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:33.631458 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="613f1277-5dc5-422e-8b92-d9be8ac858c4" containerName="istio-proxy" Nov 02 17:47:33.631607 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:33.631596 4471 cpu_manager.go:414] "containerMap: removing container" podUID="1a1b8c9e-6e63-4397-b3c0-46055331baa5" containerName="istio-proxy" Nov 02 17:47:33.631678 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:33.631669 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a1b8c9e-6e63-4397-b3c0-46055331baa5" containerName="istio-proxy" Nov 02 17:47:33.631993 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:33.631981 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a1b8c9e-6e63-4397-b3c0-46055331baa5" containerName="istio-proxy" Nov 02 17:47:33.632089 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:33.632080 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="613f1277-5dc5-422e-8b92-d9be8ac858c4" containerName="istio-proxy" Nov 02 17:47:33.637543 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:33.637518 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-f91b2d-55449f7d65-nqbd9"] Nov 02 17:47:33.637789 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:33.637622 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f91b2d-55449f7d65-nqbd9" Nov 02 17:47:33.637991 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:33.637768 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f91b2d-66d89575-sdlbq"] Nov 02 17:47:33.641617 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:33.641594 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f91b2d-66d89575-sdlbq" Nov 02 17:47:33.642427 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:33.642412 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:47:33.643354 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:33.643002 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 17:47:33.643354 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:33.643048 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" podUID="1ff9e12a-8b9c-4556-ae7a-fa3011dd516c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 17:47:33.663841 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:33.663812 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-f91b2d\"" Nov 02 17:47:33.664371 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:33.664351 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-f91b2d-dockercfg-bjc4f\"" Nov 02 17:47:33.717447 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:33.717404 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f91b2d-66d89575-sdlbq"] Nov 02 17:47:33.780171 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:33.780083 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pt5c\" (UniqueName: \"kubernetes.io/projected/6b1fbd6b-0681-4558-aa65-95e14f712af6-kube-api-access-6pt5c\") pod \"payload-pre-processing-e2e-shared-a-w0-f91b2d-66d89575-sdlbq\" (UID: \"6b1fbd6b-0681-4558-aa65-95e14f712af6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f91b2d-66d89575-sdlbq" Nov 02 17:47:33.788449 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:33.788412 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34577a2f-c05a-43ea-8f04-53eedb09fa05-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-f91b2d-55449f7d65-nqbd9\" (UID: \"34577a2f-c05a-43ea-8f04-53eedb09fa05\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f91b2d-55449f7d65-nqbd9" Nov 02 17:47:33.788791 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:33.788771 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krbds\" (UniqueName: \"kubernetes.io/projected/34577a2f-c05a-43ea-8f04-53eedb09fa05-kube-api-access-krbds\") pod \"payload-processing-e2e-shared-a-w0-f91b2d-55449f7d65-nqbd9\" (UID: \"34577a2f-c05a-43ea-8f04-53eedb09fa05\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f91b2d-55449f7d65-nqbd9" Nov 02 17:47:33.791413 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:33.791386 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b1fbd6b-0681-4558-aa65-95e14f712af6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-f91b2d-66d89575-sdlbq\" (UID: \"6b1fbd6b-0681-4558-aa65-95e14f712af6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f91b2d-66d89575-sdlbq" Nov 02 17:47:33.947154 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:33.945944 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pt5c\" (UniqueName: \"kubernetes.io/projected/6b1fbd6b-0681-4558-aa65-95e14f712af6-kube-api-access-6pt5c\") pod \"payload-pre-processing-e2e-shared-a-w0-f91b2d-66d89575-sdlbq\" (UID: \"6b1fbd6b-0681-4558-aa65-95e14f712af6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f91b2d-66d89575-sdlbq" Nov 02 17:47:33.972862 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:33.963446 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34577a2f-c05a-43ea-8f04-53eedb09fa05-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-f91b2d-55449f7d65-nqbd9\" (UID: \"34577a2f-c05a-43ea-8f04-53eedb09fa05\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f91b2d-55449f7d65-nqbd9" Nov 02 17:47:33.972862 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:33.963632 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krbds\" (UniqueName: \"kubernetes.io/projected/34577a2f-c05a-43ea-8f04-53eedb09fa05-kube-api-access-krbds\") pod \"payload-processing-e2e-shared-a-w0-f91b2d-55449f7d65-nqbd9\" (UID: \"34577a2f-c05a-43ea-8f04-53eedb09fa05\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f91b2d-55449f7d65-nqbd9" Nov 02 17:47:33.972862 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:33.963760 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b1fbd6b-0681-4558-aa65-95e14f712af6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-f91b2d-66d89575-sdlbq\" (UID: \"6b1fbd6b-0681-4558-aa65-95e14f712af6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f91b2d-66d89575-sdlbq" Nov 02 17:47:33.972862 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:33.964926 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b1fbd6b-0681-4558-aa65-95e14f712af6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-f91b2d-66d89575-sdlbq\" (UID: \"6b1fbd6b-0681-4558-aa65-95e14f712af6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f91b2d-66d89575-sdlbq" Nov 02 17:47:33.972862 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:33.965680 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34577a2f-c05a-43ea-8f04-53eedb09fa05-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-f91b2d-55449f7d65-nqbd9\" (UID: \"34577a2f-c05a-43ea-8f04-53eedb09fa05\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f91b2d-55449f7d65-nqbd9" Nov 02 17:47:34.076463 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:34.076425 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 02 17:47:34.076779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:34.076738 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" podUID="e5bb3fbe-93ce-44aa-9cf1-02394cc28af7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 02 17:47:34.196112 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:34.196067 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f6da5f-57dbdd6f67-xsfnw" event={"ID":"00044b6e-d520-4934-ba8a-b2d638f899d6","Type":"ContainerStarted","Data":"1a824126f4408c3860bd740bfa885601d050dd42ba4568c543b51f91c1ddddf0"} Nov 02 17:47:34.196451 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:34.196418 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f6da5f-57dbdd6f67-xsfnw" event={"ID":"00044b6e-d520-4934-ba8a-b2d638f899d6","Type":"ContainerStarted","Data":"d2d51cbf029e2b8255890d635215abcfa30c793a1b1101f5a4b20fea9274236f"} Nov 02 17:47:34.223380 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:34.197206 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f6da5f-57dbdd6f67-xsfnw" Nov 02 17:47:34.320794 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:34.253561 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f6da5f-57dbdd6f67-xsfnw" podStartSLOduration=3.253526784 podStartE2EDuration="3.253526784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:34.245396086 +0000 UTC m=+1071.940751584" watchObservedRunningTime="2026-11-02 17:47:34.253526784 +0000 UTC m=+1071.948882258" Nov 02 17:47:34.370329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:34.292400 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krbds\" (UniqueName: \"kubernetes.io/projected/34577a2f-c05a-43ea-8f04-53eedb09fa05-kube-api-access-krbds\") pod \"payload-processing-e2e-shared-a-w0-f91b2d-55449f7d65-nqbd9\" (UID: \"34577a2f-c05a-43ea-8f04-53eedb09fa05\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f91b2d-55449f7d65-nqbd9" Nov 02 17:47:34.370329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:34.301001 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pt5c\" (UniqueName: \"kubernetes.io/projected/6b1fbd6b-0681-4558-aa65-95e14f712af6-kube-api-access-6pt5c\") pod \"payload-pre-processing-e2e-shared-a-w0-f91b2d-66d89575-sdlbq\" (UID: \"6b1fbd6b-0681-4558-aa65-95e14f712af6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f91b2d-66d89575-sdlbq" Nov 02 17:47:34.399945 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:34.399910 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f91b2d-66d89575-sdlbq" Nov 02 17:47:34.633681 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:34.633647 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f91b2d-55449f7d65-nqbd9" Nov 02 17:47:34.711332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:34.634852 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a1b8c9e-6e63-4397-b3c0-46055331baa5" path="/var/lib/kubelet/pods/1a1b8c9e-6e63-4397-b3c0-46055331baa5/volumes" Nov 02 17:47:34.711332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:34.664400 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 17:47:34.711332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:34.696967 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" podUID="1ff9e12a-8b9c-4556-ae7a-fa3011dd516c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 17:47:34.720063 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:34.715547 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="613f1277-5dc5-422e-8b92-d9be8ac858c4" path="/var/lib/kubelet/pods/613f1277-5dc5-422e-8b92-d9be8ac858c4/volumes" Nov 02 17:47:34.720063 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:34.716946 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:47:35.066681 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:35.064179 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 02 17:47:35.066681 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:35.064231 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" podUID="e5bb3fbe-93ce-44aa-9cf1-02394cc28af7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 02 17:47:35.435136 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:47:35.435093 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b1fbd6b_0681_4558_aa65_95e14f712af6.slice/crio-c3010602331765eefcb82b20487cf8ace52cad3f9579120612572930643cb421 WatchSource:0}: Error finding container c3010602331765eefcb82b20487cf8ace52cad3f9579120612572930643cb421: Status 404 returned error can't find the container with id c3010602331765eefcb82b20487cf8ace52cad3f9579120612572930643cb421 Nov 02 17:47:35.436969 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:35.436944 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f91b2d-66d89575-sdlbq"] Nov 02 17:47:35.535009 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:47:35.534972 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34577a2f_c05a_43ea_8f04_53eedb09fa05.slice/crio-d192c51aee769e695e03c5506a590f46a94220a1382ded52156ca37d73fc7908 WatchSource:0}: Error finding container d192c51aee769e695e03c5506a590f46a94220a1382ded52156ca37d73fc7908: Status 404 returned error can't find the container with id d192c51aee769e695e03c5506a590f46a94220a1382ded52156ca37d73fc7908 Nov 02 17:47:35.535905 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:35.535602 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-f91b2d-55449f7d65-nqbd9"] Nov 02 17:47:35.642747 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:35.642719 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 17:47:35.643064 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:35.642775 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" podUID="1ff9e12a-8b9c-4556-ae7a-fa3011dd516c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 17:47:36.047886 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:36.047849 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 02 17:47:36.048069 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:36.047898 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" podUID="e5bb3fbe-93ce-44aa-9cf1-02394cc28af7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 02 17:47:36.432039 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:36.431644 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f91b2d-66d89575-sdlbq" event={"ID":"6b1fbd6b-0681-4558-aa65-95e14f712af6","Type":"ContainerStarted","Data":"891afef8e7bfd575eca7c01d2aa12528280810d7c0d04ad24a89d8aeb5a9c561"} Nov 02 17:47:36.432039 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:36.431710 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f91b2d-66d89575-sdlbq" event={"ID":"6b1fbd6b-0681-4558-aa65-95e14f712af6","Type":"ContainerStarted","Data":"c3010602331765eefcb82b20487cf8ace52cad3f9579120612572930643cb421"} Nov 02 17:47:36.432039 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:36.431959 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f91b2d-66d89575-sdlbq" Nov 02 17:47:36.441610 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:36.441576 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f91b2d-55449f7d65-nqbd9" Nov 02 17:47:36.441743 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:36.441627 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f91b2d-55449f7d65-nqbd9" event={"ID":"34577a2f-c05a-43ea-8f04-53eedb09fa05","Type":"ContainerStarted","Data":"d2a2cddf9af4db33b5d3e7cd3e37c702a3ef9e1deb08341db52b7efee7575f69"} Nov 02 17:47:36.441743 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:36.441647 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f91b2d-55449f7d65-nqbd9" event={"ID":"34577a2f-c05a-43ea-8f04-53eedb09fa05","Type":"ContainerStarted","Data":"d192c51aee769e695e03c5506a590f46a94220a1382ded52156ca37d73fc7908"} Nov 02 17:47:36.487333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:36.481676 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f91b2d-66d89575-sdlbq" podStartSLOduration=3.4816549439999998 podStartE2EDuration="3.481654944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:36.479825293 +0000 UTC m=+1074.175180793" watchObservedRunningTime="2026-11-02 17:47:36.481654944 +0000 UTC m=+1074.177010417" Nov 02 17:47:36.529609 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:36.529551 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f91b2d-55449f7d65-nqbd9" podStartSLOduration=3.529535332 podStartE2EDuration="3.529535332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:36.527053241 +0000 UTC m=+1074.222408734" watchObservedRunningTime="2026-11-02 17:47:36.529535332 +0000 UTC m=+1074.224890804" Nov 02 17:47:36.643126 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:36.643090 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 17:47:36.645327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:36.643150 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" podUID="1ff9e12a-8b9c-4556-ae7a-fa3011dd516c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 17:47:37.047157 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:37.047117 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 02 17:47:37.047378 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:37.047171 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" podUID="e5bb3fbe-93ce-44aa-9cf1-02394cc28af7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 02 17:47:37.643119 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:37.643084 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 17:47:37.643292 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:37.643135 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" podUID="1ff9e12a-8b9c-4556-ae7a-fa3011dd516c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 17:47:38.077515 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:38.077460 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 02 17:47:38.100450 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:38.100399 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" podUID="e5bb3fbe-93ce-44aa-9cf1-02394cc28af7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 02 17:47:38.659329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:38.658454 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 17:47:38.659329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:38.658520 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" podUID="1ff9e12a-8b9c-4556-ae7a-fa3011dd516c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 17:47:39.047366 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:39.047039 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 02 17:47:39.047366 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:39.047099 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" podUID="e5bb3fbe-93ce-44aa-9cf1-02394cc28af7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 02 17:47:39.215151 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:47:39.215107 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:47:39.648624 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:39.648591 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:47:39.649856 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:39.649834 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:47:40.051418 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:40.051392 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 17:47:40.051898 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:40.051883 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 17:47:40.052649 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:40.052620 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 17:47:41.734034 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:41.734006 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-504de7-786496d969-w8f9z" Nov 02 17:47:44.186433 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:44.186385 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:47:44.407331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:44.406569 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f6da5f-6cb75c9495-682jg" Nov 02 17:47:46.560624 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:46.560592 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f6da5f-57dbdd6f67-xsfnw" Nov 02 17:47:47.451707 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:47.451671 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f91b2d-55449f7d65-nqbd9" Nov 02 17:47:47.453324 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:47.453282 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f91b2d-66d89575-sdlbq" Nov 02 17:47:48.059365 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:48.059320 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:47:48.139330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:48.135421 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-7l8nw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:47:48.139330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:48.135486 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" podUID="1aa74f62-fe2f-4238-9f58-4a0a20f7892a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:47:48.367386 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:48.352941 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:47:48.400653 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:48.400589 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:47:48.426338 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:48.426284 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:47:48.537348 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:48.537286 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:47:48.565676 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:48.564563 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:47:48.566593 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:48.566552 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:47:48.727725 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:48.727622 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:47:48.794640 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:48.794605 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn"] Nov 02 17:47:48.934861 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:48.934832 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn"] Nov 02 17:47:48.935225 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:48.935205 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn" Nov 02 17:47:49.010376 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:48.962520 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-f6da5f-serving-cert\"" Nov 02 17:47:49.036332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:49.011747 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/141bdec6-5264-4a27-87e4-699a5bdcf2d7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn\" (UID: \"141bdec6-5264-4a27-87e4-699a5bdcf2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn" Nov 02 17:47:49.037219 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:49.036650 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/141bdec6-5264-4a27-87e4-699a5bdcf2d7-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn\" (UID: \"141bdec6-5264-4a27-87e4-699a5bdcf2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn" Nov 02 17:47:49.037219 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:49.036823 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/141bdec6-5264-4a27-87e4-699a5bdcf2d7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn\" (UID: \"141bdec6-5264-4a27-87e4-699a5bdcf2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn" Nov 02 17:47:49.037219 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:49.036936 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv67c\" (UniqueName: \"kubernetes.io/projected/141bdec6-5264-4a27-87e4-699a5bdcf2d7-kube-api-access-qv67c\") pod \"maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn\" (UID: \"141bdec6-5264-4a27-87e4-699a5bdcf2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn" Nov 02 17:47:49.104327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:49.071719 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:47:49.167327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:49.152286 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/141bdec6-5264-4a27-87e4-699a5bdcf2d7-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn\" (UID: \"141bdec6-5264-4a27-87e4-699a5bdcf2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn" Nov 02 17:47:49.167327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:49.152445 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/141bdec6-5264-4a27-87e4-699a5bdcf2d7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn\" (UID: \"141bdec6-5264-4a27-87e4-699a5bdcf2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn" Nov 02 17:47:49.167327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:49.152530 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv67c\" (UniqueName: \"kubernetes.io/projected/141bdec6-5264-4a27-87e4-699a5bdcf2d7-kube-api-access-qv67c\") pod \"maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn\" (UID: \"141bdec6-5264-4a27-87e4-699a5bdcf2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn" Nov 02 17:47:49.167327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:49.152625 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/141bdec6-5264-4a27-87e4-699a5bdcf2d7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn\" (UID: \"141bdec6-5264-4a27-87e4-699a5bdcf2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn" Nov 02 17:47:49.167327 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:47:49.153145 4471 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f6da5f-serving-cert: secret "maas-api-e2e-shared-b-w6-f6da5f-serving-cert" not found Nov 02 17:47:49.167327 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:47:49.153230 4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/141bdec6-5264-4a27-87e4-699a5bdcf2d7-maas-api-tls podName:141bdec6-5264-4a27-87e4-699a5bdcf2d7 nodeName:}" failed. No retries permitted until 2026-11-02 17:47:49.653208931 +0000 UTC m=+1087.348564396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/141bdec6-5264-4a27-87e4-699a5bdcf2d7-maas-api-tls") pod "maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn" (UID: "141bdec6-5264-4a27-87e4-699a5bdcf2d7") : secret "maas-api-e2e-shared-b-w6-f6da5f-serving-cert" not found Nov 02 17:47:49.168184 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:49.168124 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/141bdec6-5264-4a27-87e4-699a5bdcf2d7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn\" (UID: \"141bdec6-5264-4a27-87e4-699a5bdcf2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn" Nov 02 17:47:49.222217 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:49.220600 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/141bdec6-5264-4a27-87e4-699a5bdcf2d7-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn\" (UID: \"141bdec6-5264-4a27-87e4-699a5bdcf2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn" Nov 02 17:47:49.277396 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:49.277122 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv67c\" (UniqueName: \"kubernetes.io/projected/141bdec6-5264-4a27-87e4-699a5bdcf2d7-kube-api-access-qv67c\") pod \"maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn\" (UID: \"141bdec6-5264-4a27-87e4-699a5bdcf2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn" Nov 02 17:47:49.666968 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:49.666877 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/141bdec6-5264-4a27-87e4-699a5bdcf2d7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn\" (UID: \"141bdec6-5264-4a27-87e4-699a5bdcf2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn" Nov 02 17:47:49.669450 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:49.669419 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/141bdec6-5264-4a27-87e4-699a5bdcf2d7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn\" (UID: \"141bdec6-5264-4a27-87e4-699a5bdcf2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn" Nov 02 17:47:49.830329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:49.827624 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn" Nov 02 17:47:50.385435 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:47:50.385404 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod141bdec6_5264_4a27_87e4_699a5bdcf2d7.slice/crio-d46b239a618381b5dead56422b83078552f9c1d6633e857be018a064b60ca18b WatchSource:0}: Error finding container d46b239a618381b5dead56422b83078552f9c1d6633e857be018a064b60ca18b: Status 404 returned error can't find the container with id d46b239a618381b5dead56422b83078552f9c1d6633e857be018a064b60ca18b Nov 02 17:47:50.387020 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:50.386997 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn"] Nov 02 17:47:50.994877 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:50.994843 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn" event={"ID":"141bdec6-5264-4a27-87e4-699a5bdcf2d7","Type":"ContainerStarted","Data":"ced4e72a3b61abf8381d61a5e4e7dbed420f1aaaeb7b1aec227eaae7bb68398e"} Nov 02 17:47:50.995035 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:50.994900 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn" event={"ID":"141bdec6-5264-4a27-87e4-699a5bdcf2d7","Type":"ContainerStarted","Data":"d46b239a618381b5dead56422b83078552f9c1d6633e857be018a064b60ca18b"} Nov 02 17:47:50.995035 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:50.994929 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn" Nov 02 17:47:51.044847 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:51.044797 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn" podStartSLOduration=2.5719885639999998 podStartE2EDuration="3.044781954s" totalImagesPullingTime="472.79339ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:51.041216569 +0000 UTC m=+1088.736572064" watchObservedRunningTime="2026-11-02 17:47:51.044781954 +0000 UTC m=+1088.740137426" Nov 02 17:47:57.014569 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:47:57.014540 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn" Nov 02 17:48:09.185612 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:48:09.185578 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:48:19.565123 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:19.565089 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-c810ca-5fbdb874f9-2284p"] Nov 02 17:48:19.572062 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:19.572034 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-c810ca-5fbdb874f9-2284p" Nov 02 17:48:19.598863 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:19.598836 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-c810ca\"" Nov 02 17:48:19.720114 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:19.720080 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-c810ca-dockercfg-dq8mz\"" Nov 02 17:48:19.808402 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:19.779051 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbww9\" (UniqueName: \"kubernetes.io/projected/7395fee0-6fb5-4a90-be40-ba36fde4b163-kube-api-access-dbww9\") pod \"payload-pre-processing-e2e-shared-b-w5-c810ca-5fbdb874f9-2284p\" (UID: \"7395fee0-6fb5-4a90-be40-ba36fde4b163\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-c810ca-5fbdb874f9-2284p" Nov 02 17:48:19.808755 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:19.808727 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7395fee0-6fb5-4a90-be40-ba36fde4b163-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-c810ca-5fbdb874f9-2284p\" (UID: \"7395fee0-6fb5-4a90-be40-ba36fde4b163\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-c810ca-5fbdb874f9-2284p" Nov 02 17:48:19.992166 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:19.992065 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7395fee0-6fb5-4a90-be40-ba36fde4b163-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-c810ca-5fbdb874f9-2284p\" (UID: \"7395fee0-6fb5-4a90-be40-ba36fde4b163\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-c810ca-5fbdb874f9-2284p" Nov 02 17:48:19.993810 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:19.993778 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbww9\" (UniqueName: \"kubernetes.io/projected/7395fee0-6fb5-4a90-be40-ba36fde4b163-kube-api-access-dbww9\") pod \"payload-pre-processing-e2e-shared-b-w5-c810ca-5fbdb874f9-2284p\" (UID: \"7395fee0-6fb5-4a90-be40-ba36fde4b163\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-c810ca-5fbdb874f9-2284p" Nov 02 17:48:19.994473 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:19.993218 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7395fee0-6fb5-4a90-be40-ba36fde4b163-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-c810ca-5fbdb874f9-2284p\" (UID: \"7395fee0-6fb5-4a90-be40-ba36fde4b163\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-c810ca-5fbdb874f9-2284p" Nov 02 17:48:20.043327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:20.042280 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-c810ca-5fbdb874f9-2284p"] Nov 02 17:48:20.052543 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:20.051989 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-c810ca-7c7f98d89-lpltl"] Nov 02 17:48:20.086547 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:20.086512 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbww9\" (UniqueName: \"kubernetes.io/projected/7395fee0-6fb5-4a90-be40-ba36fde4b163-kube-api-access-dbww9\") pod \"payload-pre-processing-e2e-shared-b-w5-c810ca-5fbdb874f9-2284p\" (UID: \"7395fee0-6fb5-4a90-be40-ba36fde4b163\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-c810ca-5fbdb874f9-2284p" Nov 02 17:48:20.104791 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:20.103539 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-c810ca-7c7f98d89-lpltl"] Nov 02 17:48:20.104791 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:20.103687 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-c810ca-7c7f98d89-lpltl" Nov 02 17:48:20.187244 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:20.187209 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-c810ca-5fbdb874f9-2284p" Nov 02 17:48:20.206352 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:20.206320 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w46g8\" (UniqueName: \"kubernetes.io/projected/c3f2ca77-1f97-44a4-99f1-291daab2dee3-kube-api-access-w46g8\") pod \"payload-processing-e2e-shared-b-w5-c810ca-7c7f98d89-lpltl\" (UID: \"c3f2ca77-1f97-44a4-99f1-291daab2dee3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-c810ca-7c7f98d89-lpltl" Nov 02 17:48:20.206520 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:20.206494 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3f2ca77-1f97-44a4-99f1-291daab2dee3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-c810ca-7c7f98d89-lpltl\" (UID: \"c3f2ca77-1f97-44a4-99f1-291daab2dee3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-c810ca-7c7f98d89-lpltl" Nov 02 17:48:20.309859 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:20.308858 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w46g8\" (UniqueName: \"kubernetes.io/projected/c3f2ca77-1f97-44a4-99f1-291daab2dee3-kube-api-access-w46g8\") pod \"payload-processing-e2e-shared-b-w5-c810ca-7c7f98d89-lpltl\" (UID: \"c3f2ca77-1f97-44a4-99f1-291daab2dee3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-c810ca-7c7f98d89-lpltl" Nov 02 17:48:20.309859 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:20.308975 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3f2ca77-1f97-44a4-99f1-291daab2dee3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-c810ca-7c7f98d89-lpltl\" (UID: \"c3f2ca77-1f97-44a4-99f1-291daab2dee3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-c810ca-7c7f98d89-lpltl" Nov 02 17:48:20.309859 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:20.309807 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3f2ca77-1f97-44a4-99f1-291daab2dee3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-c810ca-7c7f98d89-lpltl\" (UID: \"c3f2ca77-1f97-44a4-99f1-291daab2dee3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-c810ca-7c7f98d89-lpltl" Nov 02 17:48:20.333385 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:20.333358 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w46g8\" (UniqueName: \"kubernetes.io/projected/c3f2ca77-1f97-44a4-99f1-291daab2dee3-kube-api-access-w46g8\") pod \"payload-processing-e2e-shared-b-w5-c810ca-7c7f98d89-lpltl\" (UID: \"c3f2ca77-1f97-44a4-99f1-291daab2dee3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-c810ca-7c7f98d89-lpltl" Nov 02 17:48:20.427359 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:20.425634 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-c810ca-7c7f98d89-lpltl" Nov 02 17:48:20.590498 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:20.590467 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-c810ca-5fbdb874f9-2284p"] Nov 02 17:48:20.590814 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:48:20.590548 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7395fee0_6fb5_4a90_be40_ba36fde4b163.slice/crio-bf902f49b92da23239e02827fc2596223836f72f2fc0ef5f18c9cfa07292f662 WatchSource:0}: Error finding container bf902f49b92da23239e02827fc2596223836f72f2fc0ef5f18c9cfa07292f662: Status 404 returned error can't find the container with id bf902f49b92da23239e02827fc2596223836f72f2fc0ef5f18c9cfa07292f662 Nov 02 17:48:20.909415 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:20.899857 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-c810ca-7c7f98d89-lpltl"] Nov 02 17:48:21.614762 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:21.614714 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-c810ca-5fbdb874f9-2284p" event={"ID":"7395fee0-6fb5-4a90-be40-ba36fde4b163","Type":"ContainerStarted","Data":"cb0c675689427d74b5951cd9f40b0265a13da471912904e4f01754fe930abe98"} Nov 02 17:48:21.615280 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:21.614795 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-c810ca-5fbdb874f9-2284p" event={"ID":"7395fee0-6fb5-4a90-be40-ba36fde4b163","Type":"ContainerStarted","Data":"bf902f49b92da23239e02827fc2596223836f72f2fc0ef5f18c9cfa07292f662"} Nov 02 17:48:21.618418 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:21.618369 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-c810ca-7c7f98d89-lpltl" event={"ID":"c3f2ca77-1f97-44a4-99f1-291daab2dee3","Type":"ContainerStarted","Data":"1ee87bfed553dc07f5edc75837272fe91b5c1c1e4a0cea934927cfbc2808dbcf"} Nov 02 17:48:21.618905 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:21.618874 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-c810ca-7c7f98d89-lpltl" event={"ID":"c3f2ca77-1f97-44a4-99f1-291daab2dee3","Type":"ContainerStarted","Data":"18c1810bfd5281493597fd9e35eef6dddd43bb0a18d4f08dce11f841db3de59d"} Nov 02 17:48:21.619381 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:21.619366 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-c810ca-5fbdb874f9-2284p" Nov 02 17:48:21.619728 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:21.619713 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-c810ca-7c7f98d89-lpltl" Nov 02 17:48:21.708331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:21.690752 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-c810ca-5fbdb874f9-2284p" podStartSLOduration=2.6907309169999998 podStartE2EDuration="2.690730917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:21.681919176 +0000 UTC m=+1119.377274648" watchObservedRunningTime="2026-11-02 17:48:21.690730917 +0000 UTC m=+1119.386086393" Nov 02 17:48:21.958801 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:21.958651 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-c810ca-7c7f98d89-lpltl" podStartSLOduration=2.9586341430000003 podStartE2EDuration="2.958634143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:21.781254143 +0000 UTC m=+1119.476609639" watchObservedRunningTime="2026-11-02 17:48:21.958634143 +0000 UTC m=+1119.653989615" Nov 02 17:48:21.961259 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:21.961226 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk"] Nov 02 17:48:21.966053 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:21.966035 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:48:21.994800 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:21.994769 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk"] Nov 02 17:48:22.013887 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:22.013859 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-workload-certs\") pod \"e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:48:22.014051 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:22.013905 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed3b7de6-bb28-433c-b507-2724a0d524df-istiod-ca-cert\") pod \"e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:48:22.014051 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:22.014020 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-istio-envoy\") pod \"e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:48:22.014163 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:22.014059 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed3b7de6-bb28-433c-b507-2724a0d524df-istio-podinfo\") pod \"e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:48:22.014271 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:22.014253 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-credential-socket\") pod \"e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:48:22.014364 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:22.014290 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6vrs\" (UniqueName: \"kubernetes.io/projected/ed3b7de6-bb28-433c-b507-2724a0d524df-kube-api-access-s6vrs\") pod \"e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:48:22.014432 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:22.014419 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-workload-socket\") pod \"e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:48:22.014487 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:22.014448 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed3b7de6-bb28-433c-b507-2724a0d524df-istio-token\") pod \"e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:48:22.014487 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:22.014479 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-istio-data\") pod \"e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:48:22.018113 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:22.017925 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-c4dbe3-openshift-default-dockercfg-96htd\"" Nov 02 17:48:22.115733 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:22.115699 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-workload-certs\") pod \"e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:48:22.115893 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:22.115745 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed3b7de6-bb28-433c-b507-2724a0d524df-istiod-ca-cert\") pod \"e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:48:22.115893 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:22.115773 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-istio-envoy\") pod \"e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:48:22.115893 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:22.115794 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed3b7de6-bb28-433c-b507-2724a0d524df-istio-podinfo\") pod \"e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:48:22.116088 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:22.115933 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-credential-socket\") pod \"e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:48:22.116088 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:22.115973 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6vrs\" (UniqueName: \"kubernetes.io/projected/ed3b7de6-bb28-433c-b507-2724a0d524df-kube-api-access-s6vrs\") pod \"e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:48:22.116088 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:22.116053 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-workload-socket\") pod \"e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:48:22.116256 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:22.116088 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed3b7de6-bb28-433c-b507-2724a0d524df-istio-token\") pod \"e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:48:22.116256 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:22.116125 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-istio-data\") pod \"e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:48:22.116404 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:22.116274 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-workload-certs\") pod \"e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:48:22.116461 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:22.116422 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-credential-socket\") pod \"e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:48:22.116653 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:22.116629 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-istio-data\") pod \"e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:48:22.116780 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:22.116733 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed3b7de6-bb28-433c-b507-2724a0d524df-istiod-ca-cert\") pod \"e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:48:22.116897 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:22.116598 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-workload-socket\") pod \"e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:48:22.118533 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:22.118512 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-istio-envoy\") pod \"e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:48:22.118941 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:22.118921 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed3b7de6-bb28-433c-b507-2724a0d524df-istio-podinfo\") pod \"e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:48:22.154995 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:22.154966 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6vrs\" (UniqueName: \"kubernetes.io/projected/ed3b7de6-bb28-433c-b507-2724a0d524df-kube-api-access-s6vrs\") pod \"e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:48:22.155114 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:22.155001 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed3b7de6-bb28-433c-b507-2724a0d524df-istio-token\") pod \"e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:48:22.284819 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:22.284790 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:48:22.862228 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:22.862190 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk"] Nov 02 17:48:23.024603 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:48:23.012268 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3b7de6_bb28_433c_b507_2724a0d524df.slice/crio-ac4300ba8c86f5039d100beed124d5683ddeecacc471fde2580b2409eee8057e WatchSource:0}: Error finding container ac4300ba8c86f5039d100beed124d5683ddeecacc471fde2580b2409eee8057e: Status 404 returned error can't find the container with id ac4300ba8c86f5039d100beed124d5683ddeecacc471fde2580b2409eee8057e Nov 02 17:48:23.031348 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:23.031285 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:48:23.031567 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:23.031545 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:48:23.031699 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:23.031682 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:48:23.738511 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:23.738472 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" event={"ID":"ed3b7de6-bb28-433c-b507-2724a0d524df","Type":"ContainerStarted","Data":"14a17f17f0639ef028abb3ff9bfc09c32e1782f4345f288ebeebbebc4c78bd31"} Nov 02 17:48:23.738677 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:23.738533 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" event={"ID":"ed3b7de6-bb28-433c-b507-2724a0d524df","Type":"ContainerStarted","Data":"ac4300ba8c86f5039d100beed124d5683ddeecacc471fde2580b2409eee8057e"} Nov 02 17:48:23.813318 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:23.813256 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" podStartSLOduration=2.813242882 podStartE2EDuration="2.813242882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:23.8111764 +0000 UTC m=+1121.506531872" watchObservedRunningTime="2026-11-02 17:48:23.813242882 +0000 UTC m=+1121.508598354" Nov 02 17:48:24.128410 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:24.127700 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-b89px"] Nov 02 17:48:24.137780 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:24.137752 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-b89px" Nov 02 17:48:24.174756 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:24.161604 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-04e682fe\"" Nov 02 17:48:24.229331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:24.219034 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e5a459e-c932-4268-9db0-5c5b007e7a72-plugins-config-volume\") pod \"payload-processing-e2e-mt-04e682fe-759db489b7-b89px\" (UID: \"7e5a459e-c932-4268-9db0-5c5b007e7a72\") " pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-b89px" Nov 02 17:48:24.229331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:24.219479 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh7qx\" (UniqueName: \"kubernetes.io/projected/7e5a459e-c932-4268-9db0-5c5b007e7a72-kube-api-access-sh7qx\") pod \"payload-processing-e2e-mt-04e682fe-759db489b7-b89px\" (UID: \"7e5a459e-c932-4268-9db0-5c5b007e7a72\") " pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-b89px" Nov 02 17:48:24.243335 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:24.240235 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-b89px"] Nov 02 17:48:24.336453 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:24.331376 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:48:24.336453 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:24.332826 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh7qx\" (UniqueName: \"kubernetes.io/projected/7e5a459e-c932-4268-9db0-5c5b007e7a72-kube-api-access-sh7qx\") pod \"payload-processing-e2e-mt-04e682fe-759db489b7-b89px\" (UID: \"7e5a459e-c932-4268-9db0-5c5b007e7a72\") " pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-b89px" Nov 02 17:48:24.336453 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:24.332908 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e5a459e-c932-4268-9db0-5c5b007e7a72-plugins-config-volume\") pod \"payload-processing-e2e-mt-04e682fe-759db489b7-b89px\" (UID: \"7e5a459e-c932-4268-9db0-5c5b007e7a72\") " pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-b89px" Nov 02 17:48:24.336453 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:24.334028 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e5a459e-c932-4268-9db0-5c5b007e7a72-plugins-config-volume\") pod \"payload-processing-e2e-mt-04e682fe-759db489b7-b89px\" (UID: \"7e5a459e-c932-4268-9db0-5c5b007e7a72\") " pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-b89px" Nov 02 17:48:24.636454 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:24.636416 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh7qx\" (UniqueName: \"kubernetes.io/projected/7e5a459e-c932-4268-9db0-5c5b007e7a72-kube-api-access-sh7qx\") pod \"payload-processing-e2e-mt-04e682fe-759db489b7-b89px\" (UID: \"7e5a459e-c932-4268-9db0-5c5b007e7a72\") " pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-b89px" Nov 02 17:48:24.779427 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:24.778423 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:48:24.803520 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:24.780390 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-b89px" Nov 02 17:48:24.837327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:24.837041 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:48:24.866345 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:24.865682 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:48:25.354519 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:25.351337 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:48:25.823368 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:25.823339 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-b89px"] Nov 02 17:48:25.846713 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:25.846684 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-b89px" event={"ID":"7e5a459e-c932-4268-9db0-5c5b007e7a72","Type":"ContainerStarted","Data":"c7761cce0528e9e39a75d2ac9a5a0d6f9f4202904284bfc1d544bfc8dbf59c37"} Nov 02 17:48:26.918332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:26.917512 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-b89px" event={"ID":"7e5a459e-c932-4268-9db0-5c5b007e7a72","Type":"ContainerStarted","Data":"09e94c315306c43e0ca811c6bae9dc775f25ef1107771c273a99e54d2be61633"} Nov 02 17:48:26.918332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:26.917644 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-b89px" Nov 02 17:48:26.981324 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:26.978728 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-b89px" podStartSLOduration=2.978708065 podStartE2EDuration="2.978708065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:26.977842992 +0000 UTC m=+1124.673198490" watchObservedRunningTime="2026-11-02 17:48:26.978708065 +0000 UTC m=+1124.674063540" Nov 02 17:48:28.649790 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:28.649754 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4dbe3-55667dc67-nnjv8"] Nov 02 17:48:28.654597 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:28.654577 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4dbe3-55667dc67-nnjv8" Nov 02 17:48:28.683325 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:28.683121 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-c4dbe3\"" Nov 02 17:48:28.715373 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:28.714992 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4dbe3-55667dc67-nnjv8"] Nov 02 17:48:28.722386 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:28.721854 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-c4dbe3-dockercfg-qp9g8\"" Nov 02 17:48:28.725010 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:28.724976 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17058cd8-b4ee-4466-807c-e47b5f98658c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-c4dbe3-55667dc67-nnjv8\" (UID: \"17058cd8-b4ee-4466-807c-e47b5f98658c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4dbe3-55667dc67-nnjv8" Nov 02 17:48:28.725749 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:28.725724 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgkrd\" (UniqueName: \"kubernetes.io/projected/17058cd8-b4ee-4466-807c-e47b5f98658c-kube-api-access-kgkrd\") pod \"payload-pre-processing-e2e-shared-b-w0-c4dbe3-55667dc67-nnjv8\" (UID: \"17058cd8-b4ee-4466-807c-e47b5f98658c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4dbe3-55667dc67-nnjv8" Nov 02 17:48:28.865329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:28.864733 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgkrd\" (UniqueName: \"kubernetes.io/projected/17058cd8-b4ee-4466-807c-e47b5f98658c-kube-api-access-kgkrd\") pod \"payload-pre-processing-e2e-shared-b-w0-c4dbe3-55667dc67-nnjv8\" (UID: \"17058cd8-b4ee-4466-807c-e47b5f98658c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4dbe3-55667dc67-nnjv8" Nov 02 17:48:28.865329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:28.864982 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17058cd8-b4ee-4466-807c-e47b5f98658c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-c4dbe3-55667dc67-nnjv8\" (UID: \"17058cd8-b4ee-4466-807c-e47b5f98658c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4dbe3-55667dc67-nnjv8" Nov 02 17:48:28.869329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:28.866321 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17058cd8-b4ee-4466-807c-e47b5f98658c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-c4dbe3-55667dc67-nnjv8\" (UID: \"17058cd8-b4ee-4466-807c-e47b5f98658c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4dbe3-55667dc67-nnjv8" Nov 02 17:48:28.967737 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:28.967643 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgkrd\" (UniqueName: \"kubernetes.io/projected/17058cd8-b4ee-4466-807c-e47b5f98658c-kube-api-access-kgkrd\") pod \"payload-pre-processing-e2e-shared-b-w0-c4dbe3-55667dc67-nnjv8\" (UID: \"17058cd8-b4ee-4466-807c-e47b5f98658c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4dbe3-55667dc67-nnjv8" Nov 02 17:48:29.178509 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:29.093584 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4dbe3-55667dc67-nnjv8" Nov 02 17:48:29.814127 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:29.813181 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4dbe3-55667dc67-nnjv8"] Nov 02 17:48:29.814127 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:48:29.813759 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17058cd8_b4ee_4466_807c_e47b5f98658c.slice/crio-3aeeeb962c3f9e23b97a859624e905ffdac6282d434dfc4b7472dd42ba58486b WatchSource:0}: Error finding container 3aeeeb962c3f9e23b97a859624e905ffdac6282d434dfc4b7472dd42ba58486b: Status 404 returned error can't find the container with id 3aeeeb962c3f9e23b97a859624e905ffdac6282d434dfc4b7472dd42ba58486b Nov 02 17:48:30.340832 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:30.340790 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4dbe3-55667dc67-nnjv8" event={"ID":"17058cd8-b4ee-4466-807c-e47b5f98658c","Type":"ContainerStarted","Data":"6ee8e9634caafdf7a729e9927a186ca34cd0e0f336cb81ef656237d228c17d59"} Nov 02 17:48:30.341003 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:30.340876 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4dbe3-55667dc67-nnjv8" event={"ID":"17058cd8-b4ee-4466-807c-e47b5f98658c","Type":"ContainerStarted","Data":"3aeeeb962c3f9e23b97a859624e905ffdac6282d434dfc4b7472dd42ba58486b"} Nov 02 17:48:30.341151 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:30.341131 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4dbe3-55667dc67-nnjv8" Nov 02 17:48:30.404955 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:30.404874 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4dbe3-55667dc67-nnjv8" podStartSLOduration=2.404855241 podStartE2EDuration="2.404855241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:30.401655103 +0000 UTC m=+1128.097010575" watchObservedRunningTime="2026-11-02 17:48:30.404855241 +0000 UTC m=+1128.100210715" Nov 02 17:48:32.733923 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:32.733889 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-c810ca-7c7f98d89-lpltl" Nov 02 17:48:32.734444 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:32.733977 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-c810ca-5fbdb874f9-2284p" Nov 02 17:48:35.434708 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:35.434627 4471 generic.go:356] "Generic (PLEG): container finished" podID="684a3084-e591-4e7c-8e0d-7f2d65211a2a" containerID="7a32f19a3d93490aca8b7b3d76882c2fbc5b96cce7557f6cd9b19f62be53f705" exitCode=0 Nov 02 17:48:35.435172 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:35.434698 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" event={"ID":"684a3084-e591-4e7c-8e0d-7f2d65211a2a","Type":"ContainerDied","Data":"7a32f19a3d93490aca8b7b3d76882c2fbc5b96cce7557f6cd9b19f62be53f705"} Nov 02 17:48:35.435249 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:35.435231 4471 scope.go:122] "RemoveContainer" containerID="7a32f19a3d93490aca8b7b3d76882c2fbc5b96cce7557f6cd9b19f62be53f705" Nov 02 17:48:35.436755 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:35.436733 4471 generic.go:356] "Generic (PLEG): container finished" podID="3de46b2c-81e7-458e-b693-088684a8a8d8" containerID="8399a59c4ce7fc6b3a252d81d472281b29c65b0724f4fe40d60c4ee606590140" exitCode=0 Nov 02 17:48:35.436841 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:35.436806 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" event={"ID":"3de46b2c-81e7-458e-b693-088684a8a8d8","Type":"ContainerDied","Data":"8399a59c4ce7fc6b3a252d81d472281b29c65b0724f4fe40d60c4ee606590140"} Nov 02 17:48:35.436991 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:35.436966 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:48:35.437048 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:35.437035 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:48:35.437084 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:35.437072 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:48:35.437213 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:35.437198 4471 scope.go:122] "RemoveContainer" containerID="8399a59c4ce7fc6b3a252d81d472281b29c65b0724f4fe40d60c4ee606590140" Nov 02 17:48:35.438884 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:35.438852 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:48:35.439023 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:35.438939 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:48:35.439023 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:35.438998 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:48:36.447291 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:36.447260 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" event={"ID":"684a3084-e591-4e7c-8e0d-7f2d65211a2a","Type":"ContainerStarted","Data":"e450181b4a251dc968cbae0bd5daa90c987f0dbf50ae3df36ae55aba977ecb3f"} Nov 02 17:48:36.449832 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:36.449808 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" event={"ID":"3de46b2c-81e7-458e-b693-088684a8a8d8","Type":"ContainerStarted","Data":"0808e5156af6b05614ccba1c65e8cd336d7aeecf8428100a36b9f35257d806f2"} Nov 02 17:48:36.623952 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:36.623923 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:48:36.624151 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:36.624116 4471 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 17:48:36.624203 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:36.624167 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" podUID="3de46b2c-81e7-458e-b693-088684a8a8d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 17:48:36.636675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:36.636651 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:48:36.636833 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:36.636815 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 02 17:48:36.636924 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:36.636909 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" podUID="684a3084-e591-4e7c-8e0d-7f2d65211a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 02 17:48:36.720944 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:36.720831 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn"] Nov 02 17:48:36.728425 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:36.728400 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:48:36.751406 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:36.751375 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn"] Nov 02 17:48:36.819666 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:36.819633 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-504de7-openshift-default-dockercfg-f4lbh\"" Nov 02 17:48:36.920704 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:36.920661 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-workload-certs\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:48:36.920887 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:36.920714 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-workload-socket\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:48:36.920887 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:36.920836 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aee26f3f-b249-457f-a1e8-77791c926ed8-istio-token\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:48:36.920990 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:36.920894 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-credential-socket\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:48:36.920990 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:36.920943 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aee26f3f-b249-457f-a1e8-77791c926ed8-istiod-ca-cert\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:48:36.920990 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:36.920972 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tpxv\" (UniqueName: \"kubernetes.io/projected/aee26f3f-b249-457f-a1e8-77791c926ed8-kube-api-access-7tpxv\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:48:36.921093 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:36.920992 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-istio-envoy\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:48:36.921093 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:36.921053 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aee26f3f-b249-457f-a1e8-77791c926ed8-istio-podinfo\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:48:36.921193 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:36.921118 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-istio-data\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:48:37.022977 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:37.022942 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aee26f3f-b249-457f-a1e8-77791c926ed8-istio-token\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:48:37.023145 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:37.023044 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-credential-socket\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:48:37.023145 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:37.023118 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aee26f3f-b249-457f-a1e8-77791c926ed8-istiod-ca-cert\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:48:37.023226 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:37.023145 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tpxv\" (UniqueName: \"kubernetes.io/projected/aee26f3f-b249-457f-a1e8-77791c926ed8-kube-api-access-7tpxv\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:48:37.023226 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:37.023174 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-istio-envoy\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:48:37.023296 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:37.023266 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aee26f3f-b249-457f-a1e8-77791c926ed8-istio-podinfo\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:48:37.023407 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:37.023388 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-istio-data\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:48:37.023449 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:37.023430 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-workload-certs\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:48:37.023483 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:37.023473 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-workload-socket\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:48:37.024030 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:37.024003 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-workload-socket\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:48:37.024182 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:37.024153 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-credential-socket\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:48:37.024236 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:37.024018 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aee26f3f-b249-457f-a1e8-77791c926ed8-istiod-ca-cert\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:48:37.024544 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:37.024517 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-workload-certs\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:48:37.024974 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:37.024580 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-istio-data\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:48:37.026279 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:37.026261 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-istio-envoy\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:48:37.026553 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:37.026534 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aee26f3f-b249-457f-a1e8-77791c926ed8-istio-podinfo\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:48:37.070070 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:37.070030 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aee26f3f-b249-457f-a1e8-77791c926ed8-istio-token\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:48:37.091278 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:37.091244 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tpxv\" (UniqueName: \"kubernetes.io/projected/aee26f3f-b249-457f-a1e8-77791c926ed8-kube-api-access-7tpxv\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:48:37.346478 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:37.346397 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:48:37.626945 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:37.625285 4471 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 17:48:37.626945 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:37.625378 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" podUID="3de46b2c-81e7-458e-b693-088684a8a8d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 17:48:37.638055 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:37.636797 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 02 17:48:37.638055 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:37.636862 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" podUID="684a3084-e591-4e7c-8e0d-7f2d65211a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 02 17:48:37.801133 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:37.801094 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn"] Nov 02 17:48:37.810752 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:48:37.810618 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaee26f3f_b249_457f_a1e8_77791c926ed8.slice/crio-97e87ea0565c73c4e95e4c506f9a0b6dc0975a3447a66a34c44f4b98b48cd6b4 WatchSource:0}: Error finding container 97e87ea0565c73c4e95e4c506f9a0b6dc0975a3447a66a34c44f4b98b48cd6b4: Status 404 returned error can't find the container with id 97e87ea0565c73c4e95e4c506f9a0b6dc0975a3447a66a34c44f4b98b48cd6b4 Nov 02 17:48:37.813603 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:37.813570 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:48:37.813697 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:37.813640 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:48:37.813697 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:37.813671 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:48:38.530018 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:38.529978 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" event={"ID":"aee26f3f-b249-457f-a1e8-77791c926ed8","Type":"ContainerStarted","Data":"622ce3377c17fef55523b5917a0ef8bd7e92084f8d70a91add39fcccf771b01a"} Nov 02 17:48:38.530291 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:38.530272 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" event={"ID":"aee26f3f-b249-457f-a1e8-77791c926ed8","Type":"ContainerStarted","Data":"97e87ea0565c73c4e95e4c506f9a0b6dc0975a3447a66a34c44f4b98b48cd6b4"} Nov 02 17:48:38.612402 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:38.611359 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" podStartSLOduration=2.611340645 podStartE2EDuration="2.611340645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:38.606648687 +0000 UTC m=+1136.302004184" watchObservedRunningTime="2026-11-02 17:48:38.611340645 +0000 UTC m=+1136.306696119" Nov 02 17:48:38.623147 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:38.623116 4471 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 17:48:38.623400 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:38.623374 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" podUID="3de46b2c-81e7-458e-b693-088684a8a8d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 17:48:38.636423 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:38.636395 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 02 17:48:38.636768 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:38.636437 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" podUID="684a3084-e591-4e7c-8e0d-7f2d65211a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 02 17:48:39.290778 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:39.289345 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-b89px" Nov 02 17:48:39.290778 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:48:39.290725 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:48:39.348469 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:39.348406 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:48:39.349552 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:39.349214 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 02 17:48:39.349552 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:39.349268 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" podUID="aee26f3f-b249-457f-a1e8-77791c926ed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 02 17:48:39.672387 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:39.655089 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 02 17:48:39.672387 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:39.655196 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" podUID="684a3084-e591-4e7c-8e0d-7f2d65211a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 02 17:48:39.672387 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:39.659575 4471 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 17:48:39.672387 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:39.659651 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" podUID="3de46b2c-81e7-458e-b693-088684a8a8d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 17:48:40.397383 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:40.392350 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 02 17:48:40.397383 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:40.392435 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" podUID="aee26f3f-b249-457f-a1e8-77791c926ed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 02 17:48:40.629327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:40.624752 4471 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 17:48:40.629327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:40.624828 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" podUID="3de46b2c-81e7-458e-b693-088684a8a8d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 17:48:40.637664 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:40.637449 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 02 17:48:40.637664 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:40.637544 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" podUID="684a3084-e591-4e7c-8e0d-7f2d65211a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 02 17:48:41.350349 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:41.348610 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 02 17:48:41.350349 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:41.348675 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" podUID="aee26f3f-b249-457f-a1e8-77791c926ed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 02 17:48:41.352669 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:41.352642 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4dbe3-55667dc67-nnjv8" Nov 02 17:48:41.681386 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:41.648740 4471 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 17:48:41.681386 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:41.648819 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" podUID="3de46b2c-81e7-458e-b693-088684a8a8d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 17:48:41.749331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:41.745107 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 02 17:48:41.749331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:41.745168 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" podUID="684a3084-e591-4e7c-8e0d-7f2d65211a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 02 17:48:42.380223 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:42.355231 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 02 17:48:42.462562 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:42.462520 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" podUID="aee26f3f-b249-457f-a1e8-77791c926ed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 02 17:48:42.696487 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:42.696402 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:48:42.699426 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:42.699403 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:48:42.701742 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:42.701721 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 17:48:42.705826 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:42.705806 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:48:42.706586 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:42.706567 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:48:42.710489 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:42.710461 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 17:48:43.363389 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:43.350544 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 02 17:48:43.363389 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:43.350620 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" podUID="aee26f3f-b249-457f-a1e8-77791c926ed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 02 17:48:44.347118 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:44.347086 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 02 17:48:44.347512 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:44.347138 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" podUID="aee26f3f-b249-457f-a1e8-77791c926ed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 02 17:48:45.352125 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:45.352093 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:48:45.352607 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:45.352587 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:48:45.353417 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:45.353398 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:48:56.016467 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:56.016417 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8"] Nov 02 17:48:56.021572 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:56.021511 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:48:56.040565 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:56.040529 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8"] Nov 02 17:48:56.132326 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:56.131588 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-f59572-openshift-default-dockercfg-5hdp8\"" Nov 02 17:48:56.255334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:56.253606 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46xsg\" (UniqueName: \"kubernetes.io/projected/85212099-c279-459d-a468-b868bb55071b-kube-api-access-46xsg\") pod \"e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:48:56.255334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:56.253664 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-istio-envoy\") pod \"e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:48:56.255334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:56.253752 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-workload-socket\") pod \"e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:48:56.255334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:56.253796 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-istio-data\") pod \"e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:48:56.255334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:56.253830 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-workload-certs\") pod \"e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:48:56.255334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:56.253917 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85212099-c279-459d-a468-b868bb55071b-istiod-ca-cert\") pod \"e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:48:56.255334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:56.253994 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85212099-c279-459d-a468-b868bb55071b-istio-podinfo\") pod \"e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:48:56.255334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:56.254040 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-credential-socket\") pod \"e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:48:56.255334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:56.254234 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85212099-c279-459d-a468-b868bb55071b-istio-token\") pod \"e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:48:56.369262 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:56.369160 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-workload-socket\") pod \"e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:48:56.375522 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:56.375489 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-istio-data\") pod \"e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:48:56.403683 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:56.403645 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-workload-certs\") pod \"e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:48:56.404017 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:56.403994 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85212099-c279-459d-a468-b868bb55071b-istiod-ca-cert\") pod \"e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:48:56.404201 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:56.404184 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85212099-c279-459d-a468-b868bb55071b-istio-podinfo\") pod \"e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:48:56.404436 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:56.404417 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-credential-socket\") pod \"e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:48:56.404784 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:56.404763 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85212099-c279-459d-a468-b868bb55071b-istio-token\") pod \"e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:48:56.405132 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:56.405107 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46xsg\" (UniqueName: \"kubernetes.io/projected/85212099-c279-459d-a468-b868bb55071b-kube-api-access-46xsg\") pod \"e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:48:56.405287 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:56.405266 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-istio-envoy\") pod \"e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:48:56.422328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:56.411753 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85212099-c279-459d-a468-b868bb55071b-istiod-ca-cert\") pod \"e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:48:56.422328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:56.369890 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-workload-socket\") pod \"e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:48:56.422328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:56.412398 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-workload-certs\") pod \"e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:48:56.422328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:56.376123 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-istio-data\") pod \"e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:48:56.422328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:56.412720 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-credential-socket\") pod \"e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:48:56.450507 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:56.442542 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-istio-envoy\") pod \"e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:48:56.536381 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:56.531321 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85212099-c279-459d-a468-b868bb55071b-istio-podinfo\") pod \"e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:48:56.568328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:56.566932 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85212099-c279-459d-a468-b868bb55071b-istio-token\") pod \"e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:48:56.616967 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:56.606328 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46xsg\" (UniqueName: \"kubernetes.io/projected/85212099-c279-459d-a468-b868bb55071b-kube-api-access-46xsg\") pod \"e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:48:56.653440 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:56.647931 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:48:57.086619 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:48:57.086589 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85212099_c279_459d_a468_b868bb55071b.slice/crio-ee6c145baa206bc871ede8a942207d664e8c1dd33bd414ead0d309f25340aeac WatchSource:0}: Error finding container ee6c145baa206bc871ede8a942207d664e8c1dd33bd414ead0d309f25340aeac: Status 404 returned error can't find the container with id ee6c145baa206bc871ede8a942207d664e8c1dd33bd414ead0d309f25340aeac Nov 02 17:48:57.089387 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:57.089352 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:48:57.089493 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:57.089396 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8"] Nov 02 17:48:57.089493 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:57.089418 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:48:57.089493 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:57.089450 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:48:57.372392 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:57.363422 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" event={"ID":"85212099-c279-459d-a468-b868bb55071b","Type":"ContainerStarted","Data":"8823091ca9765a3541591ff5624cda5fc772051fd423969892c219e35f7bfb6e"} Nov 02 17:48:57.372392 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:57.363490 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" event={"ID":"85212099-c279-459d-a468-b868bb55071b","Type":"ContainerStarted","Data":"ee6c145baa206bc871ede8a942207d664e8c1dd33bd414ead0d309f25340aeac"} Nov 02 17:48:57.446329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:57.439961 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" podStartSLOduration=2.439942155 podStartE2EDuration="2.439942155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:57.413937354 +0000 UTC m=+1155.109292847" watchObservedRunningTime="2026-11-02 17:48:57.439942155 +0000 UTC m=+1155.135297631" Nov 02 17:48:57.530329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:57.528325 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:48:57.662384 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:57.662048 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:48:58.851354 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:58.851321 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:48:59.384478 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:59.384448 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:48:59.386154 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:48:59.386128 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:49:02.347335 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:02.346454 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:49:02.507447 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:02.507392 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:49:02.577075 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:02.576394 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 02 17:49:09.185692 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:49:09.185646 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:49:16.982152 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:16.982117 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w"] Nov 02 17:49:16.982561 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:16.982350 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" podUID="59aa40f6-3083-4511-bd81-0d00135a0c7b" containerName="istio-proxy" containerID="cri-o://2246301749e89d34745b596440102ac59124843f742d7847cd002e69573bf400" gracePeriod=30 Nov 02 17:49:21.874440 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:21.874396 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4"] Nov 02 17:49:21.878893 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:21.878870 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:49:21.908834 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:21.908805 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4"] Nov 02 17:49:21.977095 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:21.977018 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istio-data\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:49:21.977290 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:21.977122 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istio-token\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:49:21.977290 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:21.977232 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-workload-certs\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:49:21.977290 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:21.977270 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rxpf\" (UniqueName: \"kubernetes.io/projected/b82b62a4-8f2a-498a-99f0-9dea59b90b07-kube-api-access-4rxpf\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:49:21.977541 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:21.977356 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istio-podinfo\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:49:21.977541 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:21.977465 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istio-envoy\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:49:21.977654 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:21.977544 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istiod-ca-cert\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:49:21.977654 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:21.977629 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-credential-socket\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:49:21.977763 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:21.977721 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-workload-socket\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:49:22.084334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.081376 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-workload-socket\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:49:22.084334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.081559 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istio-data\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:49:22.084334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.081655 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istio-token\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:49:22.084334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.081722 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-workload-certs\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:49:22.084334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.081768 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rxpf\" (UniqueName: \"kubernetes.io/projected/b82b62a4-8f2a-498a-99f0-9dea59b90b07-kube-api-access-4rxpf\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:49:22.084334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.081861 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istio-podinfo\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:49:22.084334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.081899 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istio-envoy\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:49:22.084334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.081987 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istiod-ca-cert\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:49:22.084334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.082039 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-credential-socket\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:49:22.084334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.082690 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-credential-socket\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:49:22.084334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.082948 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-workload-socket\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:49:22.084334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.083225 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istio-data\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:49:22.105408 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.099067 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-workload-certs\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:49:22.121263 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.121208 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istiod-ca-cert\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:49:22.138240 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.138024 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istio-podinfo\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:49:22.161150 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.161114 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istio-token\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:49:22.166653 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.164606 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istio-envoy\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:49:22.169242 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.168874 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rxpf\" (UniqueName: \"kubernetes.io/projected/b82b62a4-8f2a-498a-99f0-9dea59b90b07-kube-api-access-4rxpf\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:49:22.193662 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.193633 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:49:22.335253 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.335216 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:49:22.489189 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.489106 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59aa40f6-3083-4511-bd81-0d00135a0c7b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59aa40f6-3083-4511-bd81-0d00135a0c7b-istiod-ca-cert\") pod \"59aa40f6-3083-4511-bd81-0d00135a0c7b\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " Nov 02 17:49:22.489189 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.489174 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-credential-socket\") pod \"59aa40f6-3083-4511-bd81-0d00135a0c7b\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " Nov 02 17:49:22.489463 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.489243 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59aa40f6-3083-4511-bd81-0d00135a0c7b-istio-token\" (UniqueName: \"kubernetes.io/projected/59aa40f6-3083-4511-bd81-0d00135a0c7b-istio-token\") pod \"59aa40f6-3083-4511-bd81-0d00135a0c7b\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " Nov 02 17:49:22.489463 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.489363 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-istio-data\") pod \"59aa40f6-3083-4511-bd81-0d00135a0c7b\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " Nov 02 17:49:22.489463 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.489424 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-workload-socket\") pod \"59aa40f6-3083-4511-bd81-0d00135a0c7b\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " Nov 02 17:49:22.489586 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.489470 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-credential-socket" pod "59aa40f6-3083-4511-bd81-0d00135a0c7b" (UID: "59aa40f6-3083-4511-bd81-0d00135a0c7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:22.489586 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.489504 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-istio-envoy\") pod \"59aa40f6-3083-4511-bd81-0d00135a0c7b\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " Nov 02 17:49:22.489586 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.489541 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59aa40f6-3083-4511-bd81-0d00135a0c7b-istiod-ca-cert" pod "59aa40f6-3083-4511-bd81-0d00135a0c7b" (UID: "59aa40f6-3083-4511-bd81-0d00135a0c7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:49:22.489586 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.489563 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59aa40f6-3083-4511-bd81-0d00135a0c7b-kube-api-access-rfrm4\" (UniqueName: \"kubernetes.io/projected/59aa40f6-3083-4511-bd81-0d00135a0c7b-kube-api-access-rfrm4\") pod \"59aa40f6-3083-4511-bd81-0d00135a0c7b\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " Nov 02 17:49:22.489722 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.489594 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-workload-certs\") pod \"59aa40f6-3083-4511-bd81-0d00135a0c7b\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " Nov 02 17:49:22.489722 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.489684 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-istio-data" pod "59aa40f6-3083-4511-bd81-0d00135a0c7b" (UID: "59aa40f6-3083-4511-bd81-0d00135a0c7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:22.489722 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.489700 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59aa40f6-3083-4511-bd81-0d00135a0c7b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59aa40f6-3083-4511-bd81-0d00135a0c7b-istio-podinfo\") pod \"59aa40f6-3083-4511-bd81-0d00135a0c7b\" (UID: \"59aa40f6-3083-4511-bd81-0d00135a0c7b\") " Nov 02 17:49:22.489830 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.489812 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-workload-socket" pod "59aa40f6-3083-4511-bd81-0d00135a0c7b" (UID: "59aa40f6-3083-4511-bd81-0d00135a0c7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:22.489909 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.489889 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-workload-certs" pod "59aa40f6-3083-4511-bd81-0d00135a0c7b" (UID: "59aa40f6-3083-4511-bd81-0d00135a0c7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:22.490567 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.490541 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:49:22.490707 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.490571 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59aa40f6-3083-4511-bd81-0d00135a0c7b-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:49:22.490707 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.490586 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:49:22.490707 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.490601 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:49:22.490707 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.490617 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:49:22.491794 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.491766 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-istio-envoy" pod "59aa40f6-3083-4511-bd81-0d00135a0c7b" (UID: "59aa40f6-3083-4511-bd81-0d00135a0c7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:22.491952 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.491932 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59aa40f6-3083-4511-bd81-0d00135a0c7b-kube-api-access-rfrm4" pod "59aa40f6-3083-4511-bd81-0d00135a0c7b" (UID: "59aa40f6-3083-4511-bd81-0d00135a0c7b"). InnerVolumeSpecName "kube-api-access-rfrm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:49:22.492123 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.492106 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59aa40f6-3083-4511-bd81-0d00135a0c7b-istio-podinfo" pod "59aa40f6-3083-4511-bd81-0d00135a0c7b" (UID: "59aa40f6-3083-4511-bd81-0d00135a0c7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:49:22.492213 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.492198 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59aa40f6-3083-4511-bd81-0d00135a0c7b-istio-token" pod "59aa40f6-3083-4511-bd81-0d00135a0c7b" (UID: "59aa40f6-3083-4511-bd81-0d00135a0c7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:49:22.540981 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:49:22.540943 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb82b62a4_8f2a_498a_99f0_9dea59b90b07.slice/crio-761ec5437454c9c36b92b1280dc2f7f598f441d014983087d86511c9a9f1fa3b WatchSource:0}: Error finding container 761ec5437454c9c36b92b1280dc2f7f598f441d014983087d86511c9a9f1fa3b: Status 404 returned error can't find the container with id 761ec5437454c9c36b92b1280dc2f7f598f441d014983087d86511c9a9f1fa3b Nov 02 17:49:22.543525 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.543498 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4"] Nov 02 17:49:22.543937 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.543904 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:49:22.543987 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.543972 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:49:22.544030 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.544019 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:49:22.603527 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.603489 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59aa40f6-3083-4511-bd81-0d00135a0c7b-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:49:22.603527 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.603526 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59aa40f6-3083-4511-bd81-0d00135a0c7b-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:49:22.603757 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.603540 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59aa40f6-3083-4511-bd81-0d00135a0c7b-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:49:22.603757 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.603554 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfrm4\" (UniqueName: \"kubernetes.io/projected/59aa40f6-3083-4511-bd81-0d00135a0c7b-kube-api-access-rfrm4\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:49:22.846162 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.846124 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" event={"ID":"b82b62a4-8f2a-498a-99f0-9dea59b90b07","Type":"ContainerStarted","Data":"18d262b8980d88082cadbee27fa5b69bf56ee167c1900c59877069e6b06f8e6a"} Nov 02 17:49:22.846376 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.846188 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" event={"ID":"b82b62a4-8f2a-498a-99f0-9dea59b90b07","Type":"ContainerStarted","Data":"761ec5437454c9c36b92b1280dc2f7f598f441d014983087d86511c9a9f1fa3b"} Nov 02 17:49:22.847834 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.847805 4471 generic.go:356] "Generic (PLEG): container finished" podID="59aa40f6-3083-4511-bd81-0d00135a0c7b" containerID="2246301749e89d34745b596440102ac59124843f742d7847cd002e69573bf400" exitCode=0 Nov 02 17:49:22.847967 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.847844 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" Nov 02 17:49:22.847967 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.847884 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" event={"ID":"59aa40f6-3083-4511-bd81-0d00135a0c7b","Type":"ContainerDied","Data":"2246301749e89d34745b596440102ac59124843f742d7847cd002e69573bf400"} Nov 02 17:49:22.847967 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.847939 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w" event={"ID":"59aa40f6-3083-4511-bd81-0d00135a0c7b","Type":"ContainerDied","Data":"49573f62f0f8662b46ca8b7510c3f48d0b80c60a0ad6d9adc58ac365b2ad9bad"} Nov 02 17:49:22.848133 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.847967 4471 scope.go:122] "RemoveContainer" containerID="2246301749e89d34745b596440102ac59124843f742d7847cd002e69573bf400" Nov 02 17:49:22.859635 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.859606 4471 scope.go:122] "RemoveContainer" containerID="2246301749e89d34745b596440102ac59124843f742d7847cd002e69573bf400" Nov 02 17:49:22.859932 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:49:22.859913 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2246301749e89d34745b596440102ac59124843f742d7847cd002e69573bf400\": container with ID starting with 2246301749e89d34745b596440102ac59124843f742d7847cd002e69573bf400 not found: ID does not exist" containerID="2246301749e89d34745b596440102ac59124843f742d7847cd002e69573bf400" Nov 02 17:49:22.859980 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.859938 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2246301749e89d34745b596440102ac59124843f742d7847cd002e69573bf400"} err="failed to get container status \"2246301749e89d34745b596440102ac59124843f742d7847cd002e69573bf400\": rpc error: code = NotFound desc = could not find container \"2246301749e89d34745b596440102ac59124843f742d7847cd002e69573bf400\": container with ID starting with 2246301749e89d34745b596440102ac59124843f742d7847cd002e69573bf400 not found: ID does not exist" Nov 02 17:49:22.906773 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.905707 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" podStartSLOduration=1.905674447 podStartE2EDuration="1.905674447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:22.900757794 +0000 UTC m=+1180.596113271" watchObservedRunningTime="2026-11-02 17:49:22.905674447 +0000 UTC m=+1180.601029921" Nov 02 17:49:22.943433 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.943394 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w"] Nov 02 17:49:22.947261 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:22.947236 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-jgh4w"] Nov 02 17:49:23.207968 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:23.197775 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:49:23.207968 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:23.197953 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 02 17:49:23.207968 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:23.198008 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" podUID="b82b62a4-8f2a-498a-99f0-9dea59b90b07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 02 17:49:24.194053 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:24.194019 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 02 17:49:24.194555 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:24.194080 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" podUID="b82b62a4-8f2a-498a-99f0-9dea59b90b07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 02 17:49:24.447874 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:24.447790 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59aa40f6-3083-4511-bd81-0d00135a0c7b" path="/var/lib/kubelet/pods/59aa40f6-3083-4511-bd81-0d00135a0c7b/volumes" Nov 02 17:49:25.213327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:25.206703 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 02 17:49:25.213327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:25.206771 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" podUID="b82b62a4-8f2a-498a-99f0-9dea59b90b07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 02 17:49:26.223729 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:26.202026 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 02 17:49:26.223729 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:26.202137 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" podUID="b82b62a4-8f2a-498a-99f0-9dea59b90b07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 02 17:49:27.195087 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:27.195050 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 02 17:49:27.195280 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:27.195138 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" podUID="b82b62a4-8f2a-498a-99f0-9dea59b90b07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 02 17:49:28.194381 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:28.194297 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 02 17:49:28.194779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:28.194405 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" podUID="b82b62a4-8f2a-498a-99f0-9dea59b90b07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 02 17:49:29.021554 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:29.021521 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6797-0f9e42"] Nov 02 17:49:29.022340 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:29.022319 4471 cpu_manager.go:414] "containerMap: removing container" podUID="59aa40f6-3083-4511-bd81-0d00135a0c7b" containerName="istio-proxy" Nov 02 17:49:29.022340 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:29.022336 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59aa40f6-3083-4511-bd81-0d00135a0c7b" containerName="istio-proxy" Nov 02 17:49:29.022500 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:29.022491 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="59aa40f6-3083-4511-bd81-0d00135a0c7b" containerName="istio-proxy" Nov 02 17:49:29.026154 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:29.026138 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6797-0f9e42" Nov 02 17:49:29.047512 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:29.047489 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6797-0f9e42"] Nov 02 17:49:29.051494 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:29.051467 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9tmjt\"" Nov 02 17:49:29.212649 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:29.212603 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glxp5\" (UniqueName: \"kubernetes.io/projected/d07ecdd3-da66-49d7-809f-f11ebae1d6ed-kube-api-access-glxp5\") pod \"test-curl-6797-0f9e42\" (UID: \"d07ecdd3-da66-49d7-809f-f11ebae1d6ed\") " pod="opendatahub/test-curl-6797-0f9e42" Nov 02 17:49:29.213359 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:29.213291 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 02 17:49:29.213473 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:29.213375 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" podUID="b82b62a4-8f2a-498a-99f0-9dea59b90b07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 02 17:49:29.316444 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:29.316351 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glxp5\" (UniqueName: \"kubernetes.io/projected/d07ecdd3-da66-49d7-809f-f11ebae1d6ed-kube-api-access-glxp5\") pod \"test-curl-6797-0f9e42\" (UID: \"d07ecdd3-da66-49d7-809f-f11ebae1d6ed\") " pod="opendatahub/test-curl-6797-0f9e42" Nov 02 17:49:29.366957 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:29.366914 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glxp5\" (UniqueName: \"kubernetes.io/projected/d07ecdd3-da66-49d7-809f-f11ebae1d6ed-kube-api-access-glxp5\") pod \"test-curl-6797-0f9e42\" (UID: \"d07ecdd3-da66-49d7-809f-f11ebae1d6ed\") " pod="opendatahub/test-curl-6797-0f9e42" Nov 02 17:49:29.637753 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:29.637670 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6797-0f9e42" Nov 02 17:49:29.936606 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:49:29.936577 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd07ecdd3_da66_49d7_809f_f11ebae1d6ed.slice/crio-5821e7673bdc19ec24c05ce3bc4ac4026748288a0539b6cb5d3efedaef125b0f WatchSource:0}: Error finding container 5821e7673bdc19ec24c05ce3bc4ac4026748288a0539b6cb5d3efedaef125b0f: Status 404 returned error can't find the container with id 5821e7673bdc19ec24c05ce3bc4ac4026748288a0539b6cb5d3efedaef125b0f Nov 02 17:49:29.942507 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:29.942474 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6797-0f9e42"] Nov 02 17:49:30.198671 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:30.198595 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:49:30.199037 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:30.199017 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:49:30.199672 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:30.199651 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:49:31.240329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:31.235532 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6797-0f9e42" event={"ID":"d07ecdd3-da66-49d7-809f-f11ebae1d6ed","Type":"ContainerStarted","Data":"9fae759b3f7a2c4df553da75ac0841d6e6c17700ac105b786171e03c01e31ba4"} Nov 02 17:49:31.240329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:31.235583 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6797-0f9e42" event={"ID":"d07ecdd3-da66-49d7-809f-f11ebae1d6ed","Type":"ContainerStarted","Data":"5821e7673bdc19ec24c05ce3bc4ac4026748288a0539b6cb5d3efedaef125b0f"} Nov 02 17:49:31.337536 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:31.334145 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6797-0f9e42" podStartSLOduration=3.334123195 podStartE2EDuration="3.334123195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:31.303629053 +0000 UTC m=+1188.998984527" watchObservedRunningTime="2026-11-02 17:49:31.334123195 +0000 UTC m=+1189.029478670" Nov 02 17:49:32.267334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:32.260367 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6797-0f9e42"] Nov 02 17:49:32.333328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:32.324326 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6797-0f9e42"] Nov 02 17:49:32.449367 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:32.418558 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6797-28a4ce"] Nov 02 17:49:32.450279 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:32.450250 4471 cpu_manager.go:414] "containerMap: removing container" podUID="d07ecdd3-da66-49d7-809f-f11ebae1d6ed" containerName="test-curl-6797-0f9e42" Nov 02 17:49:32.450475 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:32.450459 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d07ecdd3-da66-49d7-809f-f11ebae1d6ed" containerName="test-curl-6797-0f9e42" Nov 02 17:49:32.450822 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:32.450806 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="d07ecdd3-da66-49d7-809f-f11ebae1d6ed" containerName="test-curl-6797-0f9e42" Nov 02 17:49:32.461357 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:32.457564 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6797-28a4ce" Nov 02 17:49:32.472353 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:32.472294 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6797-28a4ce"] Nov 02 17:49:32.478768 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:32.478737 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvhrc\" (UniqueName: \"kubernetes.io/projected/124388b0-e74d-44c5-8a41-963a0000e155-kube-api-access-zvhrc\") pod \"test-curl-6797-28a4ce\" (UID: \"124388b0-e74d-44c5-8a41-963a0000e155\") " pod="opendatahub/test-curl-6797-28a4ce" Nov 02 17:49:32.579980 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:32.579904 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvhrc\" (UniqueName: \"kubernetes.io/projected/124388b0-e74d-44c5-8a41-963a0000e155-kube-api-access-zvhrc\") pod \"test-curl-6797-28a4ce\" (UID: \"124388b0-e74d-44c5-8a41-963a0000e155\") " pod="opendatahub/test-curl-6797-28a4ce" Nov 02 17:49:32.610777 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:32.610756 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvhrc\" (UniqueName: \"kubernetes.io/projected/124388b0-e74d-44c5-8a41-963a0000e155-kube-api-access-zvhrc\") pod \"test-curl-6797-28a4ce\" (UID: \"124388b0-e74d-44c5-8a41-963a0000e155\") " pod="opendatahub/test-curl-6797-28a4ce" Nov 02 17:49:32.780330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:32.778318 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6797-28a4ce" Nov 02 17:49:33.245346 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:33.227263 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6797-28a4ce"] Nov 02 17:49:33.356697 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:33.345609 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6797-0f9e42" podUID="d07ecdd3-da66-49d7-809f-f11ebae1d6ed" containerName="test-curl-6797-0f9e42" containerID="cri-o://9fae759b3f7a2c4df553da75ac0841d6e6c17700ac105b786171e03c01e31ba4" gracePeriod=2 Nov 02 17:49:33.356697 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:33.345748 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6797-28a4ce" event={"ID":"124388b0-e74d-44c5-8a41-963a0000e155","Type":"ContainerStarted","Data":"c841c7ed971e4cdd767d43255a1ade1f8ecab1a0a0a18c3135b501608e2f3fea"} Nov 02 17:49:34.355583 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:34.355542 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6797-28a4ce" event={"ID":"124388b0-e74d-44c5-8a41-963a0000e155","Type":"ContainerStarted","Data":"a514856c4a6ce3e2ceccab31fd6d0a93b4cecea380f617ed5dff5525cd4ffc01"} Nov 02 17:49:34.400639 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:34.400588 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6797-28a4ce" podStartSLOduration=2.4005749659999998 podStartE2EDuration="2.400574966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:34.400480446 +0000 UTC m=+1192.095835919" watchObservedRunningTime="2026-11-02 17:49:34.400574966 +0000 UTC m=+1192.095930439" Nov 02 17:49:35.346661 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:35.346628 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6797-28a4ce"] Nov 02 17:49:35.355358 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:35.353662 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6797-28a4ce"] Nov 02 17:49:35.513203 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:35.513169 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6797-3de2fa"] Nov 02 17:49:35.514469 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:35.514449 4471 cpu_manager.go:414] "containerMap: removing container" podUID="124388b0-e74d-44c5-8a41-963a0000e155" containerName="test-curl-6797-28a4ce" Nov 02 17:49:35.514549 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:35.514472 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="124388b0-e74d-44c5-8a41-963a0000e155" containerName="test-curl-6797-28a4ce" Nov 02 17:49:35.514669 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:35.514650 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="124388b0-e74d-44c5-8a41-963a0000e155" containerName="test-curl-6797-28a4ce" Nov 02 17:49:35.518165 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:35.518145 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6797-3de2fa" Nov 02 17:49:35.532931 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:35.532906 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6797-3de2fa"] Nov 02 17:49:35.582943 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:35.582913 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbv4n\" (UniqueName: \"kubernetes.io/projected/d18b6462-514d-4ef0-b5d1-412d7afbda2d-kube-api-access-qbv4n\") pod \"test-curl-6797-3de2fa\" (UID: \"d18b6462-514d-4ef0-b5d1-412d7afbda2d\") " pod="opendatahub/test-curl-6797-3de2fa" Nov 02 17:49:35.590368 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:35.590343 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6797-0f9e42" Nov 02 17:49:35.683946 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:35.683911 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d07ecdd3-da66-49d7-809f-f11ebae1d6ed-kube-api-access-glxp5\" (UniqueName: \"kubernetes.io/projected/d07ecdd3-da66-49d7-809f-f11ebae1d6ed-kube-api-access-glxp5\") pod \"d07ecdd3-da66-49d7-809f-f11ebae1d6ed\" (UID: \"d07ecdd3-da66-49d7-809f-f11ebae1d6ed\") " Nov 02 17:49:35.684714 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:35.684687 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbv4n\" (UniqueName: \"kubernetes.io/projected/d18b6462-514d-4ef0-b5d1-412d7afbda2d-kube-api-access-qbv4n\") pod \"test-curl-6797-3de2fa\" (UID: \"d18b6462-514d-4ef0-b5d1-412d7afbda2d\") " pod="opendatahub/test-curl-6797-3de2fa" Nov 02 17:49:35.686201 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:35.686178 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d07ecdd3-da66-49d7-809f-f11ebae1d6ed-kube-api-access-glxp5" pod "d07ecdd3-da66-49d7-809f-f11ebae1d6ed" (UID: "d07ecdd3-da66-49d7-809f-f11ebae1d6ed"). InnerVolumeSpecName "kube-api-access-glxp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:49:35.716535 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:35.716510 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbv4n\" (UniqueName: \"kubernetes.io/projected/d18b6462-514d-4ef0-b5d1-412d7afbda2d-kube-api-access-qbv4n\") pod \"test-curl-6797-3de2fa\" (UID: \"d18b6462-514d-4ef0-b5d1-412d7afbda2d\") " pod="opendatahub/test-curl-6797-3de2fa" Nov 02 17:49:35.786239 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:35.786167 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glxp5\" (UniqueName: \"kubernetes.io/projected/d07ecdd3-da66-49d7-809f-f11ebae1d6ed-kube-api-access-glxp5\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:49:35.835773 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:35.835742 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6797-3de2fa" Nov 02 17:49:36.134345 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:49:36.134275 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd18b6462_514d_4ef0_b5d1_412d7afbda2d.slice/crio-6305a6a1da8646808bf71e21bd31ddc58ff026d72dde800ba2fe99025ca46d1f WatchSource:0}: Error finding container 6305a6a1da8646808bf71e21bd31ddc58ff026d72dde800ba2fe99025ca46d1f: Status 404 returned error can't find the container with id 6305a6a1da8646808bf71e21bd31ddc58ff026d72dde800ba2fe99025ca46d1f Nov 02 17:49:36.137435 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:36.137413 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6797-3de2fa"] Nov 02 17:49:36.376688 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:36.376649 4471 generic.go:356] "Generic (PLEG): container finished" podID="d07ecdd3-da66-49d7-809f-f11ebae1d6ed" containerID="9fae759b3f7a2c4df553da75ac0841d6e6c17700ac105b786171e03c01e31ba4" exitCode=137 Nov 02 17:49:36.376861 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:36.376727 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6797-0f9e42" Nov 02 17:49:36.376861 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:36.376755 4471 scope.go:122] "RemoveContainer" containerID="9fae759b3f7a2c4df553da75ac0841d6e6c17700ac105b786171e03c01e31ba4" Nov 02 17:49:36.378956 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:36.378915 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6797-3de2fa" event={"ID":"d18b6462-514d-4ef0-b5d1-412d7afbda2d","Type":"ContainerStarted","Data":"cf009b6d3bb6c1526146fd4d288844c3efe48d05d6e716f727be62ee148eaf52"} Nov 02 17:49:36.379077 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:36.378963 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6797-3de2fa" event={"ID":"d18b6462-514d-4ef0-b5d1-412d7afbda2d","Type":"ContainerStarted","Data":"6305a6a1da8646808bf71e21bd31ddc58ff026d72dde800ba2fe99025ca46d1f"} Nov 02 17:49:36.379077 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:36.379014 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6797-28a4ce" podUID="124388b0-e74d-44c5-8a41-963a0000e155" containerName="test-curl-6797-28a4ce" containerID="cri-o://a514856c4a6ce3e2ceccab31fd6d0a93b4cecea380f617ed5dff5525cd4ffc01" gracePeriod=2 Nov 02 17:49:36.387356 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:36.387339 4471 scope.go:122] "RemoveContainer" containerID="9fae759b3f7a2c4df553da75ac0841d6e6c17700ac105b786171e03c01e31ba4" Nov 02 17:49:36.387625 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:49:36.387605 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fae759b3f7a2c4df553da75ac0841d6e6c17700ac105b786171e03c01e31ba4\": container with ID starting with 9fae759b3f7a2c4df553da75ac0841d6e6c17700ac105b786171e03c01e31ba4 not found: ID does not exist" containerID="9fae759b3f7a2c4df553da75ac0841d6e6c17700ac105b786171e03c01e31ba4" Nov 02 17:49:36.387691 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:36.387637 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fae759b3f7a2c4df553da75ac0841d6e6c17700ac105b786171e03c01e31ba4"} err="failed to get container status \"9fae759b3f7a2c4df553da75ac0841d6e6c17700ac105b786171e03c01e31ba4\": rpc error: code = NotFound desc = could not find container \"9fae759b3f7a2c4df553da75ac0841d6e6c17700ac105b786171e03c01e31ba4\": container with ID starting with 9fae759b3f7a2c4df553da75ac0841d6e6c17700ac105b786171e03c01e31ba4 not found: ID does not exist" Nov 02 17:49:36.425249 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:36.425202 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6797-3de2fa" podStartSLOduration=1.425190668 podStartE2EDuration="1.425190668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:36.419953941 +0000 UTC m=+1194.115309439" watchObservedRunningTime="2026-11-02 17:49:36.425190668 +0000 UTC m=+1194.120546142" Nov 02 17:49:36.433952 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:49:36.433925 4471 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6797-28a4ce\" is forbidden: User \"system:node:ip-10-0-142-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-26.ec2.internal' and this object" podUID="124388b0-e74d-44c5-8a41-963a0000e155" pod="opendatahub/test-curl-6797-28a4ce" Nov 02 17:49:36.443163 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:36.443142 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d07ecdd3-da66-49d7-809f-f11ebae1d6ed" path="/var/lib/kubelet/pods/d07ecdd3-da66-49d7-809f-f11ebae1d6ed/volumes" Nov 02 17:49:36.448052 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:49:36.448027 4471 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6797-0f9e42\" is forbidden: User \"system:node:ip-10-0-142-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-26.ec2.internal' and this object" podUID="d07ecdd3-da66-49d7-809f-f11ebae1d6ed" pod="opendatahub/test-curl-6797-0f9e42" Nov 02 17:49:36.462130 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:49:36.462106 4471 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6797-28a4ce\" is forbidden: User \"system:node:ip-10-0-142-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-26.ec2.internal' and this object" podUID="124388b0-e74d-44c5-8a41-963a0000e155" pod="opendatahub/test-curl-6797-28a4ce" Nov 02 17:49:37.359024 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:37.358989 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6797-3de2fa"] Nov 02 17:49:37.365386 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:37.365354 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6797-3de2fa"] Nov 02 17:49:37.530425 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:37.530398 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6797-f8e9d6"] Nov 02 17:49:37.531172 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:37.531157 4471 cpu_manager.go:414] "containerMap: removing container" podUID="d18b6462-514d-4ef0-b5d1-412d7afbda2d" containerName="test-curl-6797-3de2fa" Nov 02 17:49:37.531212 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:37.531173 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d18b6462-514d-4ef0-b5d1-412d7afbda2d" containerName="test-curl-6797-3de2fa" Nov 02 17:49:37.531337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:37.531326 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="d18b6462-514d-4ef0-b5d1-412d7afbda2d" containerName="test-curl-6797-3de2fa" Nov 02 17:49:37.534179 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:37.534161 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6797-f8e9d6" Nov 02 17:49:37.550026 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:37.550004 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6797-f8e9d6"] Nov 02 17:49:37.630189 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:37.630106 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msblq\" (UniqueName: \"kubernetes.io/projected/87f39ea9-15bf-4a96-b60e-66983f3b1908-kube-api-access-msblq\") pod \"test-curl-6797-f8e9d6\" (UID: \"87f39ea9-15bf-4a96-b60e-66983f3b1908\") " pod="opendatahub/test-curl-6797-f8e9d6" Nov 02 17:49:37.732039 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:37.732004 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msblq\" (UniqueName: \"kubernetes.io/projected/87f39ea9-15bf-4a96-b60e-66983f3b1908-kube-api-access-msblq\") pod \"test-curl-6797-f8e9d6\" (UID: \"87f39ea9-15bf-4a96-b60e-66983f3b1908\") " pod="opendatahub/test-curl-6797-f8e9d6" Nov 02 17:49:37.760708 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:37.760683 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msblq\" (UniqueName: \"kubernetes.io/projected/87f39ea9-15bf-4a96-b60e-66983f3b1908-kube-api-access-msblq\") pod \"test-curl-6797-f8e9d6\" (UID: \"87f39ea9-15bf-4a96-b60e-66983f3b1908\") " pod="opendatahub/test-curl-6797-f8e9d6" Nov 02 17:49:37.846197 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:37.846168 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6797-f8e9d6" Nov 02 17:49:37.925446 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:37.925410 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:49:38.408254 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:38.408211 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6797-f8e9d6"] Nov 02 17:49:38.422365 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:38.422266 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6797-f8e9d6" event={"ID":"87f39ea9-15bf-4a96-b60e-66983f3b1908","Type":"ContainerStarted","Data":"985245cf3189567d6925f2ca79670f6b86ef96e9eada37a5130e2c472957083f"} Nov 02 17:49:38.423756 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:38.422695 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6797-3de2fa" podUID="d18b6462-514d-4ef0-b5d1-412d7afbda2d" containerName="test-curl-6797-3de2fa" containerID="cri-o://cf009b6d3bb6c1526146fd4d288844c3efe48d05d6e716f727be62ee148eaf52" gracePeriod=2 Nov 02 17:49:38.437497 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:49:38.437463 4471 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6797-3de2fa\" is forbidden: User \"system:node:ip-10-0-142-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-26.ec2.internal' and this object" podUID="d18b6462-514d-4ef0-b5d1-412d7afbda2d" pod="opendatahub/test-curl-6797-3de2fa" Nov 02 17:49:38.639013 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:38.638929 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6797-28a4ce" Nov 02 17:49:38.731695 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:38.729923 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/124388b0-e74d-44c5-8a41-963a0000e155-kube-api-access-zvhrc\" (UniqueName: \"kubernetes.io/projected/124388b0-e74d-44c5-8a41-963a0000e155-kube-api-access-zvhrc\") pod \"124388b0-e74d-44c5-8a41-963a0000e155\" (UID: \"124388b0-e74d-44c5-8a41-963a0000e155\") " Nov 02 17:49:38.745251 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:38.745209 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/124388b0-e74d-44c5-8a41-963a0000e155-kube-api-access-zvhrc" pod "124388b0-e74d-44c5-8a41-963a0000e155" (UID: "124388b0-e74d-44c5-8a41-963a0000e155"). InnerVolumeSpecName "kube-api-access-zvhrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:49:38.836678 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:38.836646 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvhrc\" (UniqueName: \"kubernetes.io/projected/124388b0-e74d-44c5-8a41-963a0000e155-kube-api-access-zvhrc\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:49:39.512808 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:39.512747 4471 generic.go:356] "Generic (PLEG): container finished" podID="124388b0-e74d-44c5-8a41-963a0000e155" containerID="a514856c4a6ce3e2ceccab31fd6d0a93b4cecea380f617ed5dff5525cd4ffc01" exitCode=137 Nov 02 17:49:39.513452 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:39.513431 4471 scope.go:122] "RemoveContainer" containerID="a514856c4a6ce3e2ceccab31fd6d0a93b4cecea380f617ed5dff5525cd4ffc01" Nov 02 17:49:39.513765 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:39.513748 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6797-28a4ce" Nov 02 17:49:39.521401 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:49:39.517953 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:49:39.534518 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:39.534489 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6797-f8e9d6" event={"ID":"87f39ea9-15bf-4a96-b60e-66983f3b1908","Type":"ContainerStarted","Data":"28d6dcbaa46d5291581a0c14b647429f45a7cf3b1f334acbb7bd668547a653af"} Nov 02 17:49:39.544800 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:39.544777 4471 scope.go:122] "RemoveContainer" containerID="a514856c4a6ce3e2ceccab31fd6d0a93b4cecea380f617ed5dff5525cd4ffc01" Nov 02 17:49:39.545469 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:49:39.545441 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a514856c4a6ce3e2ceccab31fd6d0a93b4cecea380f617ed5dff5525cd4ffc01\": container with ID starting with a514856c4a6ce3e2ceccab31fd6d0a93b4cecea380f617ed5dff5525cd4ffc01 not found: ID does not exist" containerID="a514856c4a6ce3e2ceccab31fd6d0a93b4cecea380f617ed5dff5525cd4ffc01" Nov 02 17:49:39.545563 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:39.545482 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a514856c4a6ce3e2ceccab31fd6d0a93b4cecea380f617ed5dff5525cd4ffc01"} err="failed to get container status \"a514856c4a6ce3e2ceccab31fd6d0a93b4cecea380f617ed5dff5525cd4ffc01\": rpc error: code = NotFound desc = could not find container \"a514856c4a6ce3e2ceccab31fd6d0a93b4cecea380f617ed5dff5525cd4ffc01\": container with ID starting with a514856c4a6ce3e2ceccab31fd6d0a93b4cecea380f617ed5dff5525cd4ffc01 not found: ID does not exist" Nov 02 17:49:39.547582 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:49:39.547514 4471 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6797-28a4ce\" is forbidden: User \"system:node:ip-10-0-142-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-26.ec2.internal' and this object" podUID="124388b0-e74d-44c5-8a41-963a0000e155" pod="opendatahub/test-curl-6797-28a4ce" Nov 02 17:49:39.561719 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:49:39.561691 4471 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6797-28a4ce\" is forbidden: User \"system:node:ip-10-0-142-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-26.ec2.internal' and this object" podUID="124388b0-e74d-44c5-8a41-963a0000e155" pod="opendatahub/test-curl-6797-28a4ce" Nov 02 17:49:39.667130 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:39.658999 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6797-f8e9d6" podStartSLOduration=2.658972522 podStartE2EDuration="2.658972522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:39.614794391 +0000 UTC m=+1197.310149886" watchObservedRunningTime="2026-11-02 17:49:39.658972522 +0000 UTC m=+1197.354328009" Nov 02 17:49:40.442979 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:40.442951 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="124388b0-e74d-44c5-8a41-963a0000e155" path="/var/lib/kubelet/pods/124388b0-e74d-44c5-8a41-963a0000e155/volumes" Nov 02 17:49:40.548269 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:40.548237 4471 generic.go:356] "Generic (PLEG): container finished" podID="d18b6462-514d-4ef0-b5d1-412d7afbda2d" containerID="cf009b6d3bb6c1526146fd4d288844c3efe48d05d6e716f727be62ee148eaf52" exitCode=137 Nov 02 17:49:40.850329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:40.848284 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6797-f8e9d6"] Nov 02 17:49:40.852960 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:40.852929 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6797-f8e9d6"] Nov 02 17:49:40.929287 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:40.929261 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6797-3de2fa" Nov 02 17:49:41.036514 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:41.036480 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d18b6462-514d-4ef0-b5d1-412d7afbda2d-kube-api-access-qbv4n\" (UniqueName: \"kubernetes.io/projected/d18b6462-514d-4ef0-b5d1-412d7afbda2d-kube-api-access-qbv4n\") pod \"d18b6462-514d-4ef0-b5d1-412d7afbda2d\" (UID: \"d18b6462-514d-4ef0-b5d1-412d7afbda2d\") " Nov 02 17:49:41.038706 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:41.038683 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d18b6462-514d-4ef0-b5d1-412d7afbda2d-kube-api-access-qbv4n" pod "d18b6462-514d-4ef0-b5d1-412d7afbda2d" (UID: "d18b6462-514d-4ef0-b5d1-412d7afbda2d"). InnerVolumeSpecName "kube-api-access-qbv4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:49:41.139742 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:41.139655 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbv4n\" (UniqueName: \"kubernetes.io/projected/d18b6462-514d-4ef0-b5d1-412d7afbda2d-kube-api-access-qbv4n\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:49:41.586995 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:41.586950 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6797-f8e9d6" podUID="87f39ea9-15bf-4a96-b60e-66983f3b1908" containerName="test-curl-6797-f8e9d6" containerID="cri-o://28d6dcbaa46d5291581a0c14b647429f45a7cf3b1f334acbb7bd668547a653af" gracePeriod=2 Nov 02 17:49:41.587546 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:41.587287 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6797-3de2fa" Nov 02 17:49:41.588874 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:41.588805 4471 scope.go:122] "RemoveContainer" containerID="cf009b6d3bb6c1526146fd4d288844c3efe48d05d6e716f727be62ee148eaf52" Nov 02 17:49:41.603476 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:49:41.603452 4471 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6797-f8e9d6\" is forbidden: User \"system:node:ip-10-0-142-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-26.ec2.internal' and this object" podUID="87f39ea9-15bf-4a96-b60e-66983f3b1908" pod="opendatahub/test-curl-6797-f8e9d6" Nov 02 17:49:41.617474 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:49:41.617445 4471 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6797-f8e9d6\" is forbidden: User \"system:node:ip-10-0-142-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-26.ec2.internal' and this object" podUID="87f39ea9-15bf-4a96-b60e-66983f3b1908" pod="opendatahub/test-curl-6797-f8e9d6" Nov 02 17:49:41.631571 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:49:41.631545 4471 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6797-3de2fa\" is forbidden: User \"system:node:ip-10-0-142-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-26.ec2.internal' and this object" podUID="d18b6462-514d-4ef0-b5d1-412d7afbda2d" pod="opendatahub/test-curl-6797-3de2fa" Nov 02 17:49:42.477344 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:49:42.473565 4471 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6797-f8e9d6\" is forbidden: User \"system:node:ip-10-0-142-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-26.ec2.internal' and this object" podUID="87f39ea9-15bf-4a96-b60e-66983f3b1908" pod="opendatahub/test-curl-6797-f8e9d6" Nov 02 17:49:42.491861 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:49:42.491804 4471 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6797-3de2fa\" is forbidden: User \"system:node:ip-10-0-142-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-26.ec2.internal' and this object" podUID="d18b6462-514d-4ef0-b5d1-412d7afbda2d" pod="opendatahub/test-curl-6797-3de2fa" Nov 02 17:49:42.568597 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:42.568560 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d18b6462-514d-4ef0-b5d1-412d7afbda2d" path="/var/lib/kubelet/pods/d18b6462-514d-4ef0-b5d1-412d7afbda2d/volumes" Nov 02 17:49:44.034099 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:44.034063 4471 generic.go:356] "Generic (PLEG): container finished" podID="87f39ea9-15bf-4a96-b60e-66983f3b1908" containerID="28d6dcbaa46d5291581a0c14b647429f45a7cf3b1f334acbb7bd668547a653af" exitCode=137 Nov 02 17:49:44.095758 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:44.095728 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6797-f8e9d6" Nov 02 17:49:44.212713 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:44.212680 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87f39ea9-15bf-4a96-b60e-66983f3b1908-kube-api-access-msblq\" (UniqueName: \"kubernetes.io/projected/87f39ea9-15bf-4a96-b60e-66983f3b1908-kube-api-access-msblq\") pod \"87f39ea9-15bf-4a96-b60e-66983f3b1908\" (UID: \"87f39ea9-15bf-4a96-b60e-66983f3b1908\") " Nov 02 17:49:44.215125 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:44.215092 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87f39ea9-15bf-4a96-b60e-66983f3b1908-kube-api-access-msblq" pod "87f39ea9-15bf-4a96-b60e-66983f3b1908" (UID: "87f39ea9-15bf-4a96-b60e-66983f3b1908"). InnerVolumeSpecName "kube-api-access-msblq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:49:44.315109 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:44.315029 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msblq\" (UniqueName: \"kubernetes.io/projected/87f39ea9-15bf-4a96-b60e-66983f3b1908-kube-api-access-msblq\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:49:44.443300 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:44.443259 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87f39ea9-15bf-4a96-b60e-66983f3b1908" path="/var/lib/kubelet/pods/87f39ea9-15bf-4a96-b60e-66983f3b1908/volumes" Nov 02 17:49:45.043053 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:45.043023 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6797-f8e9d6" Nov 02 17:49:45.043494 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:45.043067 4471 scope.go:122] "RemoveContainer" containerID="28d6dcbaa46d5291581a0c14b647429f45a7cf3b1f334acbb7bd668547a653af" Nov 02 17:49:51.092451 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:51.092417 4471 generic.go:356] "Generic (PLEG): container finished" podID="ed3b7de6-bb28-433c-b507-2724a0d524df" containerID="14a17f17f0639ef028abb3ff9bfc09c32e1782f4345f288ebeebbebc4c78bd31" exitCode=0 Nov 02 17:49:51.092841 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:51.092495 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" event={"ID":"ed3b7de6-bb28-433c-b507-2724a0d524df","Type":"ContainerDied","Data":"14a17f17f0639ef028abb3ff9bfc09c32e1782f4345f288ebeebbebc4c78bd31"} Nov 02 17:49:51.092921 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:51.092906 4471 scope.go:122] "RemoveContainer" containerID="14a17f17f0639ef028abb3ff9bfc09c32e1782f4345f288ebeebbebc4c78bd31" Nov 02 17:49:51.094794 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:51.094764 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:49:51.094862 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:51.094843 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:49:51.094897 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:51.094881 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:49:52.101981 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:52.101947 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" event={"ID":"ed3b7de6-bb28-433c-b507-2724a0d524df","Type":"ContainerStarted","Data":"f5b97eb650650b3a76a709d7952ecede904fbc9d15ddb7fd5a7ca1f61e3e4a41"} Nov 02 17:49:52.286034 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:52.285997 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:49:52.299408 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:52.299367 4471 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 02 17:49:52.299664 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:52.299440 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" podUID="ed3b7de6-bb28-433c-b507-2724a0d524df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 02 17:49:53.285928 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:53.285890 4471 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 02 17:49:53.286437 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:53.285951 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" podUID="ed3b7de6-bb28-433c-b507-2724a0d524df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 02 17:49:54.292328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:54.287465 4471 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 02 17:49:54.292328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:54.287535 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" podUID="ed3b7de6-bb28-433c-b507-2724a0d524df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 02 17:49:54.837171 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:54.837083 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:49:55.285996 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:55.285960 4471 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 02 17:49:55.286185 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:55.286027 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" podUID="ed3b7de6-bb28-433c-b507-2724a0d524df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 02 17:49:56.293338 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:56.291418 4471 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 02 17:49:56.293338 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:56.291489 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" podUID="ed3b7de6-bb28-433c-b507-2724a0d524df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 02 17:49:57.285222 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:57.285184 4471 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 02 17:49:57.285433 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:57.285252 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" podUID="ed3b7de6-bb28-433c-b507-2724a0d524df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 02 17:49:58.290461 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:58.290429 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:49:58.291618 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:49:58.291595 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 17:50:09.184969 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:50:09.184933 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:50:12.769116 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:12.769071 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q"] Nov 02 17:50:12.770443 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:12.770417 4471 cpu_manager.go:414] "containerMap: removing container" podUID="87f39ea9-15bf-4a96-b60e-66983f3b1908" containerName="test-curl-6797-f8e9d6" Nov 02 17:50:12.770443 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:12.770444 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87f39ea9-15bf-4a96-b60e-66983f3b1908" containerName="test-curl-6797-f8e9d6" Nov 02 17:50:12.770651 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:12.770635 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="87f39ea9-15bf-4a96-b60e-66983f3b1908" containerName="test-curl-6797-f8e9d6" Nov 02 17:50:12.774187 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:12.774160 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 17:50:12.797488 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:12.797461 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q"] Nov 02 17:50:12.804973 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:12.804942 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-2672ed-openshift-default-dockercfg-dwvdz\"" Nov 02 17:50:13.039980 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:13.039888 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-workload-certs\") pod \"e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 17:50:13.090330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:13.040343 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-istio-data\") pod \"e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 17:50:13.090639 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:13.090609 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vq4k\" (UniqueName: \"kubernetes.io/projected/c987fa61-24a5-40f2-956f-70f3947bdcd7-kube-api-access-8vq4k\") pod \"e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 17:50:13.090804 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:13.090785 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-credential-socket\") pod \"e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 17:50:13.091138 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:13.091116 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-istio-envoy\") pod \"e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 17:50:13.091349 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:13.091330 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c987fa61-24a5-40f2-956f-70f3947bdcd7-istio-podinfo\") pod \"e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 17:50:13.091564 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:13.091546 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-workload-socket\") pod \"e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 17:50:13.091709 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:13.091692 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c987fa61-24a5-40f2-956f-70f3947bdcd7-istiod-ca-cert\") pod \"e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 17:50:13.099329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:13.091952 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c987fa61-24a5-40f2-956f-70f3947bdcd7-istio-token\") pod \"e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 17:50:13.202332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:13.201226 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c987fa61-24a5-40f2-956f-70f3947bdcd7-istio-podinfo\") pod \"e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 17:50:13.202332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:13.201364 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-workload-socket\") pod \"e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 17:50:13.202332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:13.201415 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c987fa61-24a5-40f2-956f-70f3947bdcd7-istiod-ca-cert\") pod \"e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 17:50:13.202332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:13.201574 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c987fa61-24a5-40f2-956f-70f3947bdcd7-istio-token\") pod \"e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 17:50:13.202332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:13.201634 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-workload-certs\") pod \"e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 17:50:13.202332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:13.201668 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-istio-data\") pod \"e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 17:50:13.202332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:13.201706 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vq4k\" (UniqueName: \"kubernetes.io/projected/c987fa61-24a5-40f2-956f-70f3947bdcd7-kube-api-access-8vq4k\") pod \"e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 17:50:13.202332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:13.201761 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-credential-socket\") pod \"e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 17:50:13.202332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:13.201928 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-istio-envoy\") pod \"e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 17:50:13.202940 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:13.202475 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-workload-socket\") pod \"e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 17:50:13.203322 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:13.203105 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c987fa61-24a5-40f2-956f-70f3947bdcd7-istiod-ca-cert\") pod \"e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 17:50:13.229549 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:13.229511 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-credential-socket\") pod \"e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 17:50:13.231689 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:13.231665 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-workload-certs\") pod \"e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 17:50:13.232058 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:13.232038 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-istio-data\") pod \"e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 17:50:13.237176 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:13.237147 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c987fa61-24a5-40f2-956f-70f3947bdcd7-istio-podinfo\") pod \"e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 17:50:13.255528 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:13.255493 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c987fa61-24a5-40f2-956f-70f3947bdcd7-istio-token\") pod \"e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 17:50:13.257288 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:13.256353 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vq4k\" (UniqueName: \"kubernetes.io/projected/c987fa61-24a5-40f2-956f-70f3947bdcd7-kube-api-access-8vq4k\") pod \"e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 17:50:13.285329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:13.260566 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-istio-envoy\") pod \"e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 17:50:13.391865 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:13.391740 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 17:50:13.867591 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:50:13.867557 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc987fa61_24a5_40f2_956f_70f3947bdcd7.slice/crio-2a7ea84d137b91c8e68418cc5911f02452253a00e13cef1e872755fb74fe3e5e WatchSource:0}: Error finding container 2a7ea84d137b91c8e68418cc5911f02452253a00e13cef1e872755fb74fe3e5e: Status 404 returned error can't find the container with id 2a7ea84d137b91c8e68418cc5911f02452253a00e13cef1e872755fb74fe3e5e Nov 02 17:50:13.870194 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:13.870173 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q"] Nov 02 17:50:13.870670 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:13.870640 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:50:13.870723 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:13.870705 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:50:13.870759 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:13.870732 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:50:14.407046 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:14.407001 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" event={"ID":"c987fa61-24a5-40f2-956f-70f3947bdcd7","Type":"ContainerStarted","Data":"6182dc3e28ebfb10afbe89a2fd4b4998635ab1802458412842a544fad88fd2bb"} Nov 02 17:50:14.407223 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:14.407063 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" event={"ID":"c987fa61-24a5-40f2-956f-70f3947bdcd7","Type":"ContainerStarted","Data":"2a7ea84d137b91c8e68418cc5911f02452253a00e13cef1e872755fb74fe3e5e"} Nov 02 17:50:14.470945 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:14.470890 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" podStartSLOduration=2.470875472 podStartE2EDuration="2.470875472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:14.467255595 +0000 UTC m=+1232.162611087" watchObservedRunningTime="2026-11-02 17:50:14.470875472 +0000 UTC m=+1232.166230944" Nov 02 17:50:15.392137 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:15.392106 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 17:50:15.398405 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:15.398380 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 17:50:15.414390 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:15.414367 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 17:50:15.415090 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:15.415067 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 17:50:19.589735 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:19.589686 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2672ed-747f99c7d7-mm9nh"] Nov 02 17:50:19.594143 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:19.594115 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2672ed-747f99c7d7-mm9nh" Nov 02 17:50:19.602843 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:19.602808 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2672ed-747f99c7d7-mm9nh"] Nov 02 17:50:19.611102 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:19.611068 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-2672ed\"" Nov 02 17:50:19.622961 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:19.622928 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-2672ed-dockercfg-w5682\"" Nov 02 17:50:19.647876 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:19.647830 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-2672ed-69855f494c-rpbz7"] Nov 02 17:50:19.690702 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:19.690651 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2672ed-69855f494c-rpbz7" Nov 02 17:50:19.700002 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:19.699940 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqf5k\" (UniqueName: \"kubernetes.io/projected/1408655c-5efa-4883-b9d2-0b51327d40bb-kube-api-access-sqf5k\") pod \"payload-pre-processing-e2e-shared-b-w4-2672ed-747f99c7d7-mm9nh\" (UID: \"1408655c-5efa-4883-b9d2-0b51327d40bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2672ed-747f99c7d7-mm9nh" Nov 02 17:50:19.943415 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:19.942935 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1408655c-5efa-4883-b9d2-0b51327d40bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-2672ed-747f99c7d7-mm9nh\" (UID: \"1408655c-5efa-4883-b9d2-0b51327d40bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2672ed-747f99c7d7-mm9nh" Nov 02 17:50:19.943415 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:19.923405 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-2672ed-69855f494c-rpbz7"] Nov 02 17:50:20.125365 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:20.120244 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqf5k\" (UniqueName: \"kubernetes.io/projected/1408655c-5efa-4883-b9d2-0b51327d40bb-kube-api-access-sqf5k\") pod \"payload-pre-processing-e2e-shared-b-w4-2672ed-747f99c7d7-mm9nh\" (UID: \"1408655c-5efa-4883-b9d2-0b51327d40bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2672ed-747f99c7d7-mm9nh" Nov 02 17:50:20.179435 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:20.175580 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4f7g\" (UniqueName: \"kubernetes.io/projected/0381ed09-c883-448a-baf2-c7cde6ed90ab-kube-api-access-r4f7g\") pod \"payload-processing-e2e-shared-b-w4-2672ed-69855f494c-rpbz7\" (UID: \"0381ed09-c883-448a-baf2-c7cde6ed90ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2672ed-69855f494c-rpbz7" Nov 02 17:50:20.179435 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:20.175677 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1408655c-5efa-4883-b9d2-0b51327d40bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-2672ed-747f99c7d7-mm9nh\" (UID: \"1408655c-5efa-4883-b9d2-0b51327d40bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2672ed-747f99c7d7-mm9nh" Nov 02 17:50:20.179435 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:20.175874 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0381ed09-c883-448a-baf2-c7cde6ed90ab-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-2672ed-69855f494c-rpbz7\" (UID: \"0381ed09-c883-448a-baf2-c7cde6ed90ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2672ed-69855f494c-rpbz7" Nov 02 17:50:20.179435 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:20.177328 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1408655c-5efa-4883-b9d2-0b51327d40bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-2672ed-747f99c7d7-mm9nh\" (UID: \"1408655c-5efa-4883-b9d2-0b51327d40bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2672ed-747f99c7d7-mm9nh" Nov 02 17:50:20.214334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:20.214234 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqf5k\" (UniqueName: \"kubernetes.io/projected/1408655c-5efa-4883-b9d2-0b51327d40bb-kube-api-access-sqf5k\") pod \"payload-pre-processing-e2e-shared-b-w4-2672ed-747f99c7d7-mm9nh\" (UID: \"1408655c-5efa-4883-b9d2-0b51327d40bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2672ed-747f99c7d7-mm9nh" Nov 02 17:50:20.214938 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:20.214914 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2672ed-747f99c7d7-mm9nh" Nov 02 17:50:20.289330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:20.284064 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4f7g\" (UniqueName: \"kubernetes.io/projected/0381ed09-c883-448a-baf2-c7cde6ed90ab-kube-api-access-r4f7g\") pod \"payload-processing-e2e-shared-b-w4-2672ed-69855f494c-rpbz7\" (UID: \"0381ed09-c883-448a-baf2-c7cde6ed90ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2672ed-69855f494c-rpbz7" Nov 02 17:50:20.289330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:20.284286 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0381ed09-c883-448a-baf2-c7cde6ed90ab-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-2672ed-69855f494c-rpbz7\" (UID: \"0381ed09-c883-448a-baf2-c7cde6ed90ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2672ed-69855f494c-rpbz7" Nov 02 17:50:20.293902 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:20.293704 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0381ed09-c883-448a-baf2-c7cde6ed90ab-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-2672ed-69855f494c-rpbz7\" (UID: \"0381ed09-c883-448a-baf2-c7cde6ed90ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2672ed-69855f494c-rpbz7" Nov 02 17:50:20.376326 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:20.367423 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4f7g\" (UniqueName: \"kubernetes.io/projected/0381ed09-c883-448a-baf2-c7cde6ed90ab-kube-api-access-r4f7g\") pod \"payload-processing-e2e-shared-b-w4-2672ed-69855f494c-rpbz7\" (UID: \"0381ed09-c883-448a-baf2-c7cde6ed90ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2672ed-69855f494c-rpbz7" Nov 02 17:50:20.421051 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:20.421020 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2672ed-69855f494c-rpbz7" Nov 02 17:50:20.742580 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:50:20.742544 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1408655c_5efa_4883_b9d2_0b51327d40bb.slice/crio-3eb7736cf7b57a5f4634b1ad38d9444d116acdc2da4f97a7127a73f08d1ee3a3 WatchSource:0}: Error finding container 3eb7736cf7b57a5f4634b1ad38d9444d116acdc2da4f97a7127a73f08d1ee3a3: Status 404 returned error can't find the container with id 3eb7736cf7b57a5f4634b1ad38d9444d116acdc2da4f97a7127a73f08d1ee3a3 Nov 02 17:50:20.743957 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:20.743929 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2672ed-747f99c7d7-mm9nh"] Nov 02 17:50:20.788813 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:20.788771 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-2672ed-69855f494c-rpbz7"] Nov 02 17:50:21.617393 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:21.617349 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2672ed-69855f494c-rpbz7" event={"ID":"0381ed09-c883-448a-baf2-c7cde6ed90ab","Type":"ContainerStarted","Data":"a24f727007bdbf63fa47b09515fabd28a2fc5e6b81a634b7acaf960fd71af3a0"} Nov 02 17:50:21.617607 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:21.617416 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2672ed-69855f494c-rpbz7" event={"ID":"0381ed09-c883-448a-baf2-c7cde6ed90ab","Type":"ContainerStarted","Data":"cff9d81475896af4275fe9f88b4bb11cb1315ecd845dbf91bfe7f30b43cdcbce"} Nov 02 17:50:21.617607 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:21.617470 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2672ed-69855f494c-rpbz7" Nov 02 17:50:21.619167 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:21.619130 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2672ed-747f99c7d7-mm9nh" event={"ID":"1408655c-5efa-4883-b9d2-0b51327d40bb","Type":"ContainerStarted","Data":"900fb89e03912d8d0fcedd0773d6d19aa70f40a99544bee32b86f7c29df1e49c"} Nov 02 17:50:21.619285 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:21.619178 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2672ed-747f99c7d7-mm9nh" event={"ID":"1408655c-5efa-4883-b9d2-0b51327d40bb","Type":"ContainerStarted","Data":"3eb7736cf7b57a5f4634b1ad38d9444d116acdc2da4f97a7127a73f08d1ee3a3"} Nov 02 17:50:21.619285 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:21.619208 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2672ed-747f99c7d7-mm9nh" Nov 02 17:50:21.698329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:21.695968 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2672ed-69855f494c-rpbz7" podStartSLOduration=2.695948924 podStartE2EDuration="2.695948924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:21.677834203 +0000 UTC m=+1239.373189677" watchObservedRunningTime="2026-11-02 17:50:21.695948924 +0000 UTC m=+1239.391304398" Nov 02 17:50:21.766331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:21.754923 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2672ed-747f99c7d7-mm9nh" podStartSLOduration=2.75490187 podStartE2EDuration="2.75490187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:21.75253257 +0000 UTC m=+1239.447888044" watchObservedRunningTime="2026-11-02 17:50:21.75490187 +0000 UTC m=+1239.450257362" Nov 02 17:50:29.394554 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:29.394458 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 17:50:29.394554 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:29.394542 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" podUID="85212099-c279-459d-a468-b868bb55071b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 17:50:29.770808 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:29.770772 4471 generic.go:356] "Generic (PLEG): container finished" podID="85212099-c279-459d-a468-b868bb55071b" containerID="8823091ca9765a3541591ff5624cda5fc772051fd423969892c219e35f7bfb6e" exitCode=0 Nov 02 17:50:29.771002 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:29.770826 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" event={"ID":"85212099-c279-459d-a468-b868bb55071b","Type":"ContainerDied","Data":"8823091ca9765a3541591ff5624cda5fc772051fd423969892c219e35f7bfb6e"} Nov 02 17:50:29.771252 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:29.771231 4471 scope.go:122] "RemoveContainer" containerID="8823091ca9765a3541591ff5624cda5fc772051fd423969892c219e35f7bfb6e" Nov 02 17:50:29.772994 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:29.772961 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:50:29.773080 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:29.773032 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:50:29.773080 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:29.773065 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:50:30.800605 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:30.800567 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" event={"ID":"85212099-c279-459d-a468-b868bb55071b","Type":"ContainerStarted","Data":"4ca6c89eda89cedd6c28328c6c5825b82ec5d26142af68e6b0e01f39d21f6002"} Nov 02 17:50:31.649289 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:31.649258 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:50:31.649477 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:31.649458 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 17:50:31.649528 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:31.649506 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" podUID="85212099-c279-459d-a468-b868bb55071b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 17:50:32.629284 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:32.629257 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2672ed-69855f494c-rpbz7" Nov 02 17:50:32.629709 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:32.629363 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2672ed-747f99c7d7-mm9nh" Nov 02 17:50:32.648931 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:32.648906 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 17:50:32.649092 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:32.648957 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" podUID="85212099-c279-459d-a468-b868bb55071b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 17:50:33.648697 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:33.648661 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 17:50:33.649171 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:33.648713 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" podUID="85212099-c279-459d-a468-b868bb55071b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 17:50:34.648699 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:34.648664 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 17:50:34.649092 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:34.648719 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" podUID="85212099-c279-459d-a468-b868bb55071b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 17:50:35.649586 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:35.649508 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 17:50:35.649586 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:35.649560 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" podUID="85212099-c279-459d-a468-b868bb55071b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 17:50:36.685256 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:36.679588 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:50:36.685256 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:36.684809 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:50:36.701987 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:36.701854 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 17:50:39.365541 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:50:39.365501 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:50:47.455681 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:47.455647 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg"] Nov 02 17:50:47.456196 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:47.455834 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" podUID="60f62cc3-c3a5-4d48-995f-4844ae54e13a" containerName="istio-proxy" containerID="cri-o://65e88f15f027979be09a4e14b8eec60e3f4c85cb6d234be3f5cd4e5ed37a6d7e" gracePeriod=30 Nov 02 17:50:47.468299 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:47.468256 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk"] Nov 02 17:50:47.468565 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:47.468525 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" podUID="913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324" containerName="istio-proxy" containerID="cri-o://db5da7d150d9657a69000f52f423a6e997e39321395780cf57a27248785e73e9" gracePeriod=30 Nov 02 17:50:47.473187 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:47.473155 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg"] Nov 02 17:50:47.473414 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:47.473387 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" podUID="85637eda-7caa-4bc0-b697-1acf7f1da917" containerName="istio-proxy" containerID="cri-o://e0df35339cd046c1b1729013cf72507ab8d8f5c36842e269aba20629304b77b8" gracePeriod=30 Nov 02 17:50:48.471371 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:48.471338 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55"] Nov 02 17:50:48.475960 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:48.475936 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:50:48.502588 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:48.502558 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55"] Nov 02 17:50:48.511356 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:48.507958 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-04e682fe-openshift-default-dockercfg-2whl8\"" Nov 02 17:50:48.618003 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:48.617967 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-workload-socket\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:50:48.618203 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:48.618180 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d505d54d-e785-4f07-87c1-7aef49f974a3-istiod-ca-cert\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:50:48.618388 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:48.618369 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2b4h\" (UniqueName: \"kubernetes.io/projected/d505d54d-e785-4f07-87c1-7aef49f974a3-kube-api-access-n2b4h\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:50:48.618478 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:48.618403 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-istio-envoy\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:50:48.618565 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:48.618550 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d505d54d-e785-4f07-87c1-7aef49f974a3-istio-podinfo\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:50:48.618624 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:48.618579 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d505d54d-e785-4f07-87c1-7aef49f974a3-istio-token\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:50:48.618664 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:48.618654 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-workload-certs\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:50:48.618699 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:48.618685 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-credential-socket\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:50:48.618738 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:48.618718 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-istio-data\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:50:48.720478 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:48.720445 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2b4h\" (UniqueName: \"kubernetes.io/projected/d505d54d-e785-4f07-87c1-7aef49f974a3-kube-api-access-n2b4h\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:50:48.720664 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:48.720484 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-istio-envoy\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:50:48.720664 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:48.720559 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d505d54d-e785-4f07-87c1-7aef49f974a3-istio-podinfo\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:50:48.720664 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:48.720578 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d505d54d-e785-4f07-87c1-7aef49f974a3-istio-token\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:50:48.720664 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:48.720607 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-workload-certs\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:50:48.720664 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:48.720641 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-credential-socket\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:50:48.720935 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:48.720687 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-istio-data\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:50:48.720935 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:48.720764 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-workload-socket\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:50:48.720935 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:48.720929 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d505d54d-e785-4f07-87c1-7aef49f974a3-istiod-ca-cert\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:50:48.721125 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:48.721090 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-istio-data\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:50:48.721208 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:48.721102 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-credential-socket\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:50:48.721412 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:48.721343 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-workload-certs\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:50:48.721412 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:48.721388 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-workload-socket\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:50:48.721763 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:48.721737 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d505d54d-e785-4f07-87c1-7aef49f974a3-istiod-ca-cert\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:50:48.723010 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:48.722988 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-istio-envoy\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:50:48.723368 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:48.723348 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d505d54d-e785-4f07-87c1-7aef49f974a3-istio-podinfo\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:50:48.746095 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:48.746070 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2b4h\" (UniqueName: \"kubernetes.io/projected/d505d54d-e785-4f07-87c1-7aef49f974a3-kube-api-access-n2b4h\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:50:48.746191 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:48.746104 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d505d54d-e785-4f07-87c1-7aef49f974a3-istio-token\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:50:48.788847 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:48.788817 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:50:49.108973 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:50:49.108936 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd505d54d_e785_4f07_87c1_7aef49f974a3.slice/crio-7bf96e026ea64cb01a811a8608e4f3539c25fa7cc0e5055c19100a1dc1f21324 WatchSource:0}: Error finding container 7bf96e026ea64cb01a811a8608e4f3539c25fa7cc0e5055c19100a1dc1f21324: Status 404 returned error can't find the container with id 7bf96e026ea64cb01a811a8608e4f3539c25fa7cc0e5055c19100a1dc1f21324 Nov 02 17:50:49.112208 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:49.112181 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55"] Nov 02 17:50:49.112340 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:49.112214 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:50:49.112340 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:49.112287 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:50:49.112436 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:49.112368 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:50:49.148960 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:49.148932 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" event={"ID":"d505d54d-e785-4f07-87c1-7aef49f974a3","Type":"ContainerStarted","Data":"7bf96e026ea64cb01a811a8608e4f3539c25fa7cc0e5055c19100a1dc1f21324"} Nov 02 17:50:50.223339 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:50.223277 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" event={"ID":"d505d54d-e785-4f07-87c1-7aef49f974a3","Type":"ContainerStarted","Data":"be9bb81b215f788abfab429f970bb3c8b80255b51a78eab8169717feee0c5673"} Nov 02 17:50:50.789846 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:50.789807 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:50:50.791849 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:50.791817 4471 patch_prober.go:29] interesting pod/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 17:50:50.792008 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:50.791873 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" podUID="d505d54d-e785-4f07-87c1-7aef49f974a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 17:50:51.789874 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:51.789841 4471 patch_prober.go:29] interesting pod/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 17:50:51.790275 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:51.789896 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" podUID="d505d54d-e785-4f07-87c1-7aef49f974a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 17:50:52.789666 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.789570 4471 patch_prober.go:29] interesting pod/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 17:50:52.790010 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.789968 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" podUID="d505d54d-e785-4f07-87c1-7aef49f974a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 17:50:52.851772 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.851736 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:50:52.912205 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.912176 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:50:52.923878 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.923844 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:50:52.935975 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.935941 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istio-podinfo\") pod \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " Nov 02 17:50:52.936121 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.936022 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-workload-certs\") pod \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " Nov 02 17:50:52.936121 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.936100 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-credential-socket\") pod \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " Nov 02 17:50:52.936244 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.936185 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-workload-certs\") pod \"85637eda-7caa-4bc0-b697-1acf7f1da917\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " Nov 02 17:50:52.936584 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.936556 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-workload-certs" pod "913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324" (UID: "913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:52.936827 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.936779 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-credential-socket\") pod \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " Nov 02 17:50:52.936952 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.936854 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-workload-certs\") pod \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " Nov 02 17:50:52.936952 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.936921 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85637eda-7caa-4bc0-b697-1acf7f1da917-kube-api-access-6z6pt\" (UniqueName: \"kubernetes.io/projected/85637eda-7caa-4bc0-b697-1acf7f1da917-kube-api-access-6z6pt\") pod \"85637eda-7caa-4bc0-b697-1acf7f1da917\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " Nov 02 17:50:52.937108 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.937007 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-istio-data\") pod \"85637eda-7caa-4bc0-b697-1acf7f1da917\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " Nov 02 17:50:52.937187 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.937143 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istio-data\") pod \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " Nov 02 17:50:52.937246 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.937223 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-workload-socket\") pod \"85637eda-7caa-4bc0-b697-1acf7f1da917\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " Nov 02 17:50:52.937297 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.937283 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60f62cc3-c3a5-4d48-995f-4844ae54e13a-kube-api-access-z82px\" (UniqueName: \"kubernetes.io/projected/60f62cc3-c3a5-4d48-995f-4844ae54e13a-kube-api-access-z82px\") pod \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " Nov 02 17:50:52.937440 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.937411 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85637eda-7caa-4bc0-b697-1acf7f1da917-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85637eda-7caa-4bc0-b697-1acf7f1da917-istiod-ca-cert\") pod \"85637eda-7caa-4bc0-b697-1acf7f1da917\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " Nov 02 17:50:52.937499 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.937454 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istio-envoy\") pod \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " Nov 02 17:50:52.937551 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.937526 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istio-envoy\") pod \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " Nov 02 17:50:52.937606 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.937585 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85637eda-7caa-4bc0-b697-1acf7f1da917-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85637eda-7caa-4bc0-b697-1acf7f1da917-istio-podinfo\") pod \"85637eda-7caa-4bc0-b697-1acf7f1da917\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " Nov 02 17:50:52.937656 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.937623 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istiod-ca-cert\") pod \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " Nov 02 17:50:52.937710 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.937674 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-credential-socket\") pod \"85637eda-7caa-4bc0-b697-1acf7f1da917\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " Nov 02 17:50:52.937836 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.937817 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-istio-envoy\") pod \"85637eda-7caa-4bc0-b697-1acf7f1da917\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " Nov 02 17:50:52.937906 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.937861 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-workload-socket\") pod \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " Nov 02 17:50:52.937963 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.937914 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85637eda-7caa-4bc0-b697-1acf7f1da917-istio-token\" (UniqueName: \"kubernetes.io/projected/85637eda-7caa-4bc0-b697-1acf7f1da917-istio-token\") pod \"85637eda-7caa-4bc0-b697-1acf7f1da917\" (UID: \"85637eda-7caa-4bc0-b697-1acf7f1da917\") " Nov 02 17:50:52.938021 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.938003 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istio-token\" (UniqueName: \"kubernetes.io/projected/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istio-token\") pod \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " Nov 02 17:50:52.938273 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.938247 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-credential-socket" pod "60f62cc3-c3a5-4d48-995f-4844ae54e13a" (UID: "60f62cc3-c3a5-4d48-995f-4844ae54e13a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:52.938410 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.938348 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-kube-api-access-kgdhp\" (UniqueName: \"kubernetes.io/projected/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-kube-api-access-kgdhp\") pod \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " Nov 02 17:50:52.938554 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.938534 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istio-podinfo\") pod \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " Nov 02 17:50:52.938696 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.938681 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istio-data\") pod \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " Nov 02 17:50:52.938840 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.938825 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istio-token\" (UniqueName: \"kubernetes.io/projected/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istio-token\") pod \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " Nov 02 17:50:52.941106 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.941085 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istiod-ca-cert\") pod \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\" (UID: \"60f62cc3-c3a5-4d48-995f-4844ae54e13a\") " Nov 02 17:50:52.942391 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.942368 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-workload-socket\") pod \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\" (UID: \"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324\") " Nov 02 17:50:52.943661 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.943637 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:50:52.944326 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.944281 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:50:52.944669 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.938929 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-workload-certs" pod "85637eda-7caa-4bc0-b697-1acf7f1da917" (UID: "85637eda-7caa-4bc0-b697-1acf7f1da917"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:52.944669 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.939522 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-workload-certs" pod "60f62cc3-c3a5-4d48-995f-4844ae54e13a" (UID: "60f62cc3-c3a5-4d48-995f-4844ae54e13a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:52.944669 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.939582 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-credential-socket" pod "913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324" (UID: "913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:52.944669 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.939586 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istio-data" pod "60f62cc3-c3a5-4d48-995f-4844ae54e13a" (UID: "60f62cc3-c3a5-4d48-995f-4844ae54e13a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:52.944669 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.941082 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istio-data" pod "913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324" (UID: "913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:52.944669 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.941360 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-istio-data" pod "85637eda-7caa-4bc0-b697-1acf7f1da917" (UID: "85637eda-7caa-4bc0-b697-1acf7f1da917"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:52.944669 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.941961 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-workload-socket" pod "60f62cc3-c3a5-4d48-995f-4844ae54e13a" (UID: "60f62cc3-c3a5-4d48-995f-4844ae54e13a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:52.945822 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.942228 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-workload-socket" pod "85637eda-7caa-4bc0-b697-1acf7f1da917" (UID: "85637eda-7caa-4bc0-b697-1acf7f1da917"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:52.945822 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.942285 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istiod-ca-cert" pod "60f62cc3-c3a5-4d48-995f-4844ae54e13a" (UID: "60f62cc3-c3a5-4d48-995f-4844ae54e13a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:50:52.945822 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.942282 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-credential-socket" pod "85637eda-7caa-4bc0-b697-1acf7f1da917" (UID: "85637eda-7caa-4bc0-b697-1acf7f1da917"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:52.945822 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.943163 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istiod-ca-cert" pod "913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324" (UID: "913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:50:52.945822 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.944230 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-workload-socket" pod "913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324" (UID: "913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:52.947744 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.946619 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85637eda-7caa-4bc0-b697-1acf7f1da917-kube-api-access-6z6pt" pod "85637eda-7caa-4bc0-b697-1acf7f1da917" (UID: "85637eda-7caa-4bc0-b697-1acf7f1da917"). InnerVolumeSpecName "kube-api-access-6z6pt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:50:52.947744 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.946961 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85637eda-7caa-4bc0-b697-1acf7f1da917-istiod-ca-cert" pod "85637eda-7caa-4bc0-b697-1acf7f1da917" (UID: "85637eda-7caa-4bc0-b697-1acf7f1da917"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:50:52.947744 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.947090 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60f62cc3-c3a5-4d48-995f-4844ae54e13a-kube-api-access-z82px" pod "60f62cc3-c3a5-4d48-995f-4844ae54e13a" (UID: "60f62cc3-c3a5-4d48-995f-4844ae54e13a"). InnerVolumeSpecName "kube-api-access-z82px". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:50:52.948562 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.948280 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-kube-api-access-kgdhp" pod "913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324" (UID: "913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324"). InnerVolumeSpecName "kube-api-access-kgdhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:50:52.953389 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.950271 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istio-envoy" pod "913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324" (UID: "913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:52.953389 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.950532 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istio-envoy" pod "60f62cc3-c3a5-4d48-995f-4844ae54e13a" (UID: "60f62cc3-c3a5-4d48-995f-4844ae54e13a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:52.953389 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.953128 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" podStartSLOduration=4.953109778 podStartE2EDuration="4.953109778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:50.294151086 +0000 UTC m=+1267.989506560" watchObservedRunningTime="2026-11-02 17:50:52.953109778 +0000 UTC m=+1270.648465252" Nov 02 17:50:52.953827 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.953787 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85637eda-7caa-4bc0-b697-1acf7f1da917-istio-podinfo" pod "85637eda-7caa-4bc0-b697-1acf7f1da917" (UID: "85637eda-7caa-4bc0-b697-1acf7f1da917"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:50:52.954506 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.954481 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85637eda-7caa-4bc0-b697-1acf7f1da917-istio-token" pod "85637eda-7caa-4bc0-b697-1acf7f1da917" (UID: "85637eda-7caa-4bc0-b697-1acf7f1da917"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:50:52.954632 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.954601 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-istio-envoy" pod "85637eda-7caa-4bc0-b697-1acf7f1da917" (UID: "85637eda-7caa-4bc0-b697-1acf7f1da917"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:52.954994 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.954953 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istio-podinfo" pod "913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324" (UID: "913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:50:52.955216 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.955187 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istio-podinfo" pod "60f62cc3-c3a5-4d48-995f-4844ae54e13a" (UID: "60f62cc3-c3a5-4d48-995f-4844ae54e13a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:50:52.982578 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.982527 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istio-token" pod "60f62cc3-c3a5-4d48-995f-4844ae54e13a" (UID: "60f62cc3-c3a5-4d48-995f-4844ae54e13a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:50:52.982927 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:52.982892 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istio-token" pod "913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324" (UID: "913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:50:53.046888 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.046847 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:50:53.046888 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.046893 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:50:53.047081 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.046908 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:50:53.047081 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.046923 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z82px\" (UniqueName: \"kubernetes.io/projected/60f62cc3-c3a5-4d48-995f-4844ae54e13a-kube-api-access-z82px\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:50:53.047081 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.046938 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85637eda-7caa-4bc0-b697-1acf7f1da917-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:50:53.047081 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.046952 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:50:53.047081 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.046965 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:50:53.047081 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.046980 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85637eda-7caa-4bc0-b697-1acf7f1da917-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:50:53.047081 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.046993 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:50:53.047081 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.047005 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:50:53.047081 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.047020 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:50:53.047081 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.047034 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:50:53.047081 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.047046 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85637eda-7caa-4bc0-b697-1acf7f1da917-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:50:53.047081 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.047060 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:50:53.047081 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.047073 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgdhp\" (UniqueName: \"kubernetes.io/projected/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-kube-api-access-kgdhp\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:50:53.047490 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.047086 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:50:53.047490 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.047100 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:50:53.047490 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.047114 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:50:53.047490 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.047127 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:50:53.047490 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.047142 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:50:53.047490 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.047158 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60f62cc3-c3a5-4d48-995f-4844ae54e13a-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:50:53.047490 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.047171 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85637eda-7caa-4bc0-b697-1acf7f1da917-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:50:53.047490 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.047183 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:50:53.047490 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.047217 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60f62cc3-c3a5-4d48-995f-4844ae54e13a-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:50:53.047490 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.047231 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z6pt\" (UniqueName: \"kubernetes.io/projected/85637eda-7caa-4bc0-b697-1acf7f1da917-kube-api-access-6z6pt\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:50:53.288425 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.288389 4471 generic.go:356] "Generic (PLEG): container finished" podID="913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324" containerID="db5da7d150d9657a69000f52f423a6e997e39321395780cf57a27248785e73e9" exitCode=0 Nov 02 17:50:53.288597 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.288475 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" Nov 02 17:50:53.288597 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.288505 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" event={"ID":"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324","Type":"ContainerDied","Data":"db5da7d150d9657a69000f52f423a6e997e39321395780cf57a27248785e73e9"} Nov 02 17:50:53.288597 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.288591 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk" event={"ID":"913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324","Type":"ContainerDied","Data":"81a130821a26cf3a4b916e360ddc6ceeb810745b8a7bbd934737caeea443bcd7"} Nov 02 17:50:53.288737 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.288624 4471 scope.go:122] "RemoveContainer" containerID="db5da7d150d9657a69000f52f423a6e997e39321395780cf57a27248785e73e9" Nov 02 17:50:53.290299 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.290270 4471 generic.go:356] "Generic (PLEG): container finished" podID="60f62cc3-c3a5-4d48-995f-4844ae54e13a" containerID="65e88f15f027979be09a4e14b8eec60e3f4c85cb6d234be3f5cd4e5ed37a6d7e" exitCode=0 Nov 02 17:50:53.290299 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.290325 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" Nov 02 17:50:53.290501 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.290342 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" event={"ID":"60f62cc3-c3a5-4d48-995f-4844ae54e13a","Type":"ContainerDied","Data":"65e88f15f027979be09a4e14b8eec60e3f4c85cb6d234be3f5cd4e5ed37a6d7e"} Nov 02 17:50:53.290501 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.290428 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg" event={"ID":"60f62cc3-c3a5-4d48-995f-4844ae54e13a","Type":"ContainerDied","Data":"41bfe1babb911a4343d21ddb16d9ac56a67f3b1e8f166b85711254ee0c92b858"} Nov 02 17:50:53.292165 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.292145 4471 generic.go:356] "Generic (PLEG): container finished" podID="85637eda-7caa-4bc0-b697-1acf7f1da917" containerID="e0df35339cd046c1b1729013cf72507ab8d8f5c36842e269aba20629304b77b8" exitCode=0 Nov 02 17:50:53.292264 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.292195 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" Nov 02 17:50:53.292264 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.292223 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" event={"ID":"85637eda-7caa-4bc0-b697-1acf7f1da917","Type":"ContainerDied","Data":"e0df35339cd046c1b1729013cf72507ab8d8f5c36842e269aba20629304b77b8"} Nov 02 17:50:53.292396 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.292289 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" event={"ID":"85637eda-7caa-4bc0-b697-1acf7f1da917","Type":"ContainerDied","Data":"47d77fac58e0ffc25db7b7a1f37107ddf77ff76a88e46c4fb39489a1880afd0a"} Nov 02 17:50:53.301858 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.301735 4471 scope.go:122] "RemoveContainer" containerID="db5da7d150d9657a69000f52f423a6e997e39321395780cf57a27248785e73e9" Nov 02 17:50:53.302050 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:50:53.302033 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db5da7d150d9657a69000f52f423a6e997e39321395780cf57a27248785e73e9\": container with ID starting with db5da7d150d9657a69000f52f423a6e997e39321395780cf57a27248785e73e9 not found: ID does not exist" containerID="db5da7d150d9657a69000f52f423a6e997e39321395780cf57a27248785e73e9" Nov 02 17:50:53.302117 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.302059 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db5da7d150d9657a69000f52f423a6e997e39321395780cf57a27248785e73e9"} err="failed to get container status \"db5da7d150d9657a69000f52f423a6e997e39321395780cf57a27248785e73e9\": rpc error: code = NotFound desc = could not find container \"db5da7d150d9657a69000f52f423a6e997e39321395780cf57a27248785e73e9\": container with ID starting with db5da7d150d9657a69000f52f423a6e997e39321395780cf57a27248785e73e9 not found: ID does not exist" Nov 02 17:50:53.302117 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.302078 4471 scope.go:122] "RemoveContainer" containerID="65e88f15f027979be09a4e14b8eec60e3f4c85cb6d234be3f5cd4e5ed37a6d7e" Nov 02 17:50:53.312493 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.312476 4471 scope.go:122] "RemoveContainer" containerID="65e88f15f027979be09a4e14b8eec60e3f4c85cb6d234be3f5cd4e5ed37a6d7e" Nov 02 17:50:53.312776 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:50:53.312758 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65e88f15f027979be09a4e14b8eec60e3f4c85cb6d234be3f5cd4e5ed37a6d7e\": container with ID starting with 65e88f15f027979be09a4e14b8eec60e3f4c85cb6d234be3f5cd4e5ed37a6d7e not found: ID does not exist" containerID="65e88f15f027979be09a4e14b8eec60e3f4c85cb6d234be3f5cd4e5ed37a6d7e" Nov 02 17:50:53.312835 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.312784 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65e88f15f027979be09a4e14b8eec60e3f4c85cb6d234be3f5cd4e5ed37a6d7e"} err="failed to get container status \"65e88f15f027979be09a4e14b8eec60e3f4c85cb6d234be3f5cd4e5ed37a6d7e\": rpc error: code = NotFound desc = could not find container \"65e88f15f027979be09a4e14b8eec60e3f4c85cb6d234be3f5cd4e5ed37a6d7e\": container with ID starting with 65e88f15f027979be09a4e14b8eec60e3f4c85cb6d234be3f5cd4e5ed37a6d7e not found: ID does not exist" Nov 02 17:50:53.312835 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.312801 4471 scope.go:122] "RemoveContainer" containerID="e0df35339cd046c1b1729013cf72507ab8d8f5c36842e269aba20629304b77b8" Nov 02 17:50:53.322202 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.322186 4471 scope.go:122] "RemoveContainer" containerID="e0df35339cd046c1b1729013cf72507ab8d8f5c36842e269aba20629304b77b8" Nov 02 17:50:53.322535 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:50:53.322509 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0df35339cd046c1b1729013cf72507ab8d8f5c36842e269aba20629304b77b8\": container with ID starting with e0df35339cd046c1b1729013cf72507ab8d8f5c36842e269aba20629304b77b8 not found: ID does not exist" containerID="e0df35339cd046c1b1729013cf72507ab8d8f5c36842e269aba20629304b77b8" Nov 02 17:50:53.322600 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.322543 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0df35339cd046c1b1729013cf72507ab8d8f5c36842e269aba20629304b77b8"} err="failed to get container status \"e0df35339cd046c1b1729013cf72507ab8d8f5c36842e269aba20629304b77b8\": rpc error: code = NotFound desc = could not find container \"e0df35339cd046c1b1729013cf72507ab8d8f5c36842e269aba20629304b77b8\": container with ID starting with e0df35339cd046c1b1729013cf72507ab8d8f5c36842e269aba20629304b77b8 not found: ID does not exist" Nov 02 17:50:53.389462 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.389430 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg"] Nov 02 17:50:53.401574 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.401546 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg"] Nov 02 17:50:53.427161 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.427136 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg"] Nov 02 17:50:53.431601 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.431581 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-bq7vg"] Nov 02 17:50:53.481745 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.481678 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk"] Nov 02 17:50:53.488056 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.488023 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-gtrbk"] Nov 02 17:50:53.789846 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.789807 4471 patch_prober.go:29] interesting pod/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 17:50:53.790051 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.789874 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" podUID="d505d54d-e785-4f07-87c1-7aef49f974a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 17:50:53.826451 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.826418 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:50:53.826595 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:53.826472 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-h6cpg" podUID="85637eda-7caa-4bc0-b697-1acf7f1da917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:50:54.443147 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:54.443115 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60f62cc3-c3a5-4d48-995f-4844ae54e13a" path="/var/lib/kubelet/pods/60f62cc3-c3a5-4d48-995f-4844ae54e13a/volumes" Nov 02 17:50:54.443670 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:54.443655 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85637eda-7caa-4bc0-b697-1acf7f1da917" path="/var/lib/kubelet/pods/85637eda-7caa-4bc0-b697-1acf7f1da917/volumes" Nov 02 17:50:54.444152 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:54.444137 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324" path="/var/lib/kubelet/pods/913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324/volumes" Nov 02 17:50:54.789952 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:54.789915 4471 patch_prober.go:29] interesting pod/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 17:50:54.790130 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:54.789968 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" podUID="d505d54d-e785-4f07-87c1-7aef49f974a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 17:50:55.789778 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:55.789741 4471 patch_prober.go:29] interesting pod/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 17:50:55.789928 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:55.789797 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" podUID="d505d54d-e785-4f07-87c1-7aef49f974a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 17:50:56.794628 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:56.794596 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:50:56.795083 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:56.795059 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:50:56.795765 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:50:56.795745 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:51:00.362338 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:00.361127 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:51:02.539101 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:02.539068 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m"] Nov 02 17:51:02.539549 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:02.539360 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" podUID="0316e437-096a-4330-b71c-3b33b139c4d9" containerName="istio-proxy" containerID="cri-o://e9e0d421d895df66f634f9cc802c2666a0495c29040e2d6f268e87b5da3d400b" gracePeriod=30 Nov 02 17:51:02.552783 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:02.552752 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl"] Nov 02 17:51:02.553001 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:02.552973 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" podUID="74fa14f5-bd49-49aa-88ef-fd812f99e7bc" containerName="istio-proxy" containerID="cri-o://5f421836c954b905f591ab4fa3eb3f6f382bc4f91242e59962f16f1fe931cdc9" gracePeriod=30 Nov 02 17:51:02.557288 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:02.557261 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp"] Nov 02 17:51:02.557483 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:02.557463 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" podUID="7bec43b9-5e3f-4c4b-b435-7926df2d53cc" containerName="istio-proxy" containerID="cri-o://e692d256494448ad494843d34287ef5a4da85965a85eb81ff10633112e8e5142" gracePeriod=30 Nov 02 17:51:02.566386 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:02.566362 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286"] Nov 02 17:51:02.566774 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:02.566742 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" podUID="65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c" containerName="istio-proxy" containerID="cri-o://680fb021801780d34675eb6036f760230f7b7e1fe854273780e7d450df04379e" gracePeriod=30 Nov 02 17:51:07.672147 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:07.672106 4471 generic.go:356] "Generic (PLEG): container finished" podID="0316e437-096a-4330-b71c-3b33b139c4d9" containerID="e9e0d421d895df66f634f9cc802c2666a0495c29040e2d6f268e87b5da3d400b" exitCode=0 Nov 02 17:51:07.672668 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:07.672164 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" event={"ID":"0316e437-096a-4330-b71c-3b33b139c4d9","Type":"ContainerDied","Data":"e9e0d421d895df66f634f9cc802c2666a0495c29040e2d6f268e87b5da3d400b"} Nov 02 17:51:07.887116 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:07.887082 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:51:07.920092 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:07.920065 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:51:07.958636 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:07.958605 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0316e437-096a-4330-b71c-3b33b139c4d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0316e437-096a-4330-b71c-3b33b139c4d9-istiod-ca-cert\") pod \"0316e437-096a-4330-b71c-3b33b139c4d9\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " Nov 02 17:51:07.958818 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:07.958691 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-istio-envoy\") pod \"0316e437-096a-4330-b71c-3b33b139c4d9\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " Nov 02 17:51:07.958818 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:07.958745 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-istio-data\") pod \"0316e437-096a-4330-b71c-3b33b139c4d9\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " Nov 02 17:51:07.958948 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:07.958870 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0316e437-096a-4330-b71c-3b33b139c4d9-kube-api-access-8kls7\" (UniqueName: \"kubernetes.io/projected/0316e437-096a-4330-b71c-3b33b139c4d9-kube-api-access-8kls7\") pod \"0316e437-096a-4330-b71c-3b33b139c4d9\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " Nov 02 17:51:07.959011 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:07.958988 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-workload-certs\") pod \"0316e437-096a-4330-b71c-3b33b139c4d9\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " Nov 02 17:51:07.959106 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:07.959071 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-workload-socket\") pod \"0316e437-096a-4330-b71c-3b33b139c4d9\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " Nov 02 17:51:07.959106 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:07.959083 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0316e437-096a-4330-b71c-3b33b139c4d9-istiod-ca-cert" pod "0316e437-096a-4330-b71c-3b33b139c4d9" (UID: "0316e437-096a-4330-b71c-3b33b139c4d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:07.959275 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:07.959104 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0316e437-096a-4330-b71c-3b33b139c4d9-istio-token\" (UniqueName: \"kubernetes.io/projected/0316e437-096a-4330-b71c-3b33b139c4d9-istio-token\") pod \"0316e437-096a-4330-b71c-3b33b139c4d9\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " Nov 02 17:51:07.959366 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:07.959346 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-credential-socket\") pod \"0316e437-096a-4330-b71c-3b33b139c4d9\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " Nov 02 17:51:07.959424 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:07.959381 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-istio-data" pod "0316e437-096a-4330-b71c-3b33b139c4d9" (UID: "0316e437-096a-4330-b71c-3b33b139c4d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:07.959424 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:07.959413 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0316e437-096a-4330-b71c-3b33b139c4d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0316e437-096a-4330-b71c-3b33b139c4d9-istio-podinfo\") pod \"0316e437-096a-4330-b71c-3b33b139c4d9\" (UID: \"0316e437-096a-4330-b71c-3b33b139c4d9\") " Nov 02 17:51:07.962350 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:07.959702 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-workload-socket" pod "0316e437-096a-4330-b71c-3b33b139c4d9" (UID: "0316e437-096a-4330-b71c-3b33b139c4d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:07.962350 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:07.960282 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0316e437-096a-4330-b71c-3b33b139c4d9-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:07.962350 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:07.960326 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:07.962350 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:07.960340 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:07.962825 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:07.962650 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-credential-socket" pod "0316e437-096a-4330-b71c-3b33b139c4d9" (UID: "0316e437-096a-4330-b71c-3b33b139c4d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:07.962825 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:07.962747 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0316e437-096a-4330-b71c-3b33b139c4d9-istio-podinfo" pod "0316e437-096a-4330-b71c-3b33b139c4d9" (UID: "0316e437-096a-4330-b71c-3b33b139c4d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:51:07.962825 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:07.962802 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-workload-certs" pod "0316e437-096a-4330-b71c-3b33b139c4d9" (UID: "0316e437-096a-4330-b71c-3b33b139c4d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:07.964124 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:07.964057 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0316e437-096a-4330-b71c-3b33b139c4d9-kube-api-access-8kls7" pod "0316e437-096a-4330-b71c-3b33b139c4d9" (UID: "0316e437-096a-4330-b71c-3b33b139c4d9"). InnerVolumeSpecName "kube-api-access-8kls7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:07.964124 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:07.964101 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0316e437-096a-4330-b71c-3b33b139c4d9-istio-token" pod "0316e437-096a-4330-b71c-3b33b139c4d9" (UID: "0316e437-096a-4330-b71c-3b33b139c4d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:07.970209 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:07.970181 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-istio-envoy" pod "0316e437-096a-4330-b71c-3b33b139c4d9" (UID: "0316e437-096a-4330-b71c-3b33b139c4d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:08.003516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.003482 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:51:08.037431 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.037398 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:51:08.061601 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.061568 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istio-data\") pod \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " Nov 02 17:51:08.063236 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.061637 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-workload-socket\") pod \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " Nov 02 17:51:08.063236 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.061687 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istio-token\" (UniqueName: \"kubernetes.io/projected/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istio-token\") pod \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " Nov 02 17:51:08.063236 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.061729 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istio-data\") pod \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " Nov 02 17:51:08.063236 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.061765 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-kube-api-access-dl6v2\" (UniqueName: \"kubernetes.io/projected/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-kube-api-access-dl6v2\") pod \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " Nov 02 17:51:08.063236 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.061796 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-kube-api-access-9sm9n\" (UniqueName: \"kubernetes.io/projected/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-kube-api-access-9sm9n\") pod \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " Nov 02 17:51:08.063236 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.061832 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istiod-ca-cert\") pod \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " Nov 02 17:51:08.063236 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.061869 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istio-token\" (UniqueName: \"kubernetes.io/projected/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istio-token\") pod \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " Nov 02 17:51:08.063236 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.062931 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-credential-socket\") pod \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " Nov 02 17:51:08.063236 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.063109 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istio-podinfo\") pod \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " Nov 02 17:51:08.063236 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.063170 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istio-data" pod "7bec43b9-5e3f-4c4b-b435-7926df2d53cc" (UID: "7bec43b9-5e3f-4c4b-b435-7926df2d53cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:08.063236 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.063179 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istio-envoy\") pod \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " Nov 02 17:51:08.063892 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.063348 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istio-podinfo\") pod \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " Nov 02 17:51:08.063892 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.063403 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-workload-certs\") pod \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " Nov 02 17:51:08.063892 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.063527 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istio-envoy\") pod \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " Nov 02 17:51:08.063892 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.063622 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-workload-certs\") pod \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " Nov 02 17:51:08.063892 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.063660 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istiod-ca-cert\") pod \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\" (UID: \"74fa14f5-bd49-49aa-88ef-fd812f99e7bc\") " Nov 02 17:51:08.063892 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.063688 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-credential-socket\") pod \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " Nov 02 17:51:08.063892 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.063732 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-workload-socket\") pod \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\" (UID: \"7bec43b9-5e3f-4c4b-b435-7926df2d53cc\") " Nov 02 17:51:08.069811 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.064173 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istio-data" pod "74fa14f5-bd49-49aa-88ef-fd812f99e7bc" (UID: "74fa14f5-bd49-49aa-88ef-fd812f99e7bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:08.069811 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.064830 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:08.069811 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.064853 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0316e437-096a-4330-b71c-3b33b139c4d9-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:08.069811 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.064867 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:08.069811 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.064878 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:08.069811 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.064890 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:08.069811 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.064903 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kls7\" (UniqueName: \"kubernetes.io/projected/0316e437-096a-4330-b71c-3b33b139c4d9-kube-api-access-8kls7\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:08.069811 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.064916 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0316e437-096a-4330-b71c-3b33b139c4d9-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:08.069811 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.064927 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0316e437-096a-4330-b71c-3b33b139c4d9-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:08.069811 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.065529 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-credential-socket" pod "74fa14f5-bd49-49aa-88ef-fd812f99e7bc" (UID: "74fa14f5-bd49-49aa-88ef-fd812f99e7bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:08.069811 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.066439 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istiod-ca-cert" pod "7bec43b9-5e3f-4c4b-b435-7926df2d53cc" (UID: "7bec43b9-5e3f-4c4b-b435-7926df2d53cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:08.069811 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.066584 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istio-token" pod "7bec43b9-5e3f-4c4b-b435-7926df2d53cc" (UID: "7bec43b9-5e3f-4c4b-b435-7926df2d53cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:08.069811 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.067186 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-workload-certs" pod "7bec43b9-5e3f-4c4b-b435-7926df2d53cc" (UID: "7bec43b9-5e3f-4c4b-b435-7926df2d53cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:08.069811 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.067522 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-credential-socket" pod "7bec43b9-5e3f-4c4b-b435-7926df2d53cc" (UID: "7bec43b9-5e3f-4c4b-b435-7926df2d53cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:08.069811 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.067545 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-workload-certs" pod "74fa14f5-bd49-49aa-88ef-fd812f99e7bc" (UID: "74fa14f5-bd49-49aa-88ef-fd812f99e7bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:08.069811 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.068006 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-workload-socket" pod "74fa14f5-bd49-49aa-88ef-fd812f99e7bc" (UID: "74fa14f5-bd49-49aa-88ef-fd812f99e7bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:08.069811 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.068056 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istiod-ca-cert" pod "74fa14f5-bd49-49aa-88ef-fd812f99e7bc" (UID: "74fa14f5-bd49-49aa-88ef-fd812f99e7bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:08.069811 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.068182 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-workload-socket" pod "7bec43b9-5e3f-4c4b-b435-7926df2d53cc" (UID: "7bec43b9-5e3f-4c4b-b435-7926df2d53cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:08.069811 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.068492 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istio-podinfo" pod "7bec43b9-5e3f-4c4b-b435-7926df2d53cc" (UID: "7bec43b9-5e3f-4c4b-b435-7926df2d53cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:51:08.071402 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.069825 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-kube-api-access-9sm9n" pod "7bec43b9-5e3f-4c4b-b435-7926df2d53cc" (UID: "7bec43b9-5e3f-4c4b-b435-7926df2d53cc"). InnerVolumeSpecName "kube-api-access-9sm9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:08.071942 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.071917 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-kube-api-access-dl6v2" pod "74fa14f5-bd49-49aa-88ef-fd812f99e7bc" (UID: "74fa14f5-bd49-49aa-88ef-fd812f99e7bc"). InnerVolumeSpecName "kube-api-access-dl6v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:08.072035 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.071989 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istio-envoy" pod "74fa14f5-bd49-49aa-88ef-fd812f99e7bc" (UID: "74fa14f5-bd49-49aa-88ef-fd812f99e7bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:08.072675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.072650 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istio-podinfo" pod "74fa14f5-bd49-49aa-88ef-fd812f99e7bc" (UID: "74fa14f5-bd49-49aa-88ef-fd812f99e7bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:51:08.073389 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.073293 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istio-token" pod "74fa14f5-bd49-49aa-88ef-fd812f99e7bc" (UID: "74fa14f5-bd49-49aa-88ef-fd812f99e7bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:08.074344 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.074318 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istio-envoy" pod "7bec43b9-5e3f-4c4b-b435-7926df2d53cc" (UID: "7bec43b9-5e3f-4c4b-b435-7926df2d53cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:08.165968 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.165871 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istio-envoy\") pod \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " Nov 02 17:51:08.165968 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.165952 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-kube-api-access-chgjx\" (UniqueName: \"kubernetes.io/projected/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-kube-api-access-chgjx\") pod \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " Nov 02 17:51:08.166210 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.166013 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istio-podinfo\") pod \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " Nov 02 17:51:08.166210 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.166088 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istio-data\") pod \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " Nov 02 17:51:08.166210 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.166123 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-workload-socket\") pod \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " Nov 02 17:51:08.166427 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.166244 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-credential-socket\") pod \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " Nov 02 17:51:08.166427 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.166373 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istiod-ca-cert\") pod \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " Nov 02 17:51:08.166545 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.166452 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-workload-certs\") pod \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " Nov 02 17:51:08.166545 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.166479 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istio-token\" (UniqueName: \"kubernetes.io/projected/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istio-token\") pod \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\" (UID: \"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c\") " Nov 02 17:51:08.166664 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.166633 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-workload-socket" pod "65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c" (UID: "65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:08.167910 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.166891 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istio-data" pod "65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c" (UID: "65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:08.167910 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.166819 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-credential-socket" pod "65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c" (UID: "65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:08.167910 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.166989 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-workload-certs" pod "65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c" (UID: "65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:08.167910 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.167620 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:08.167910 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.167648 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:08.167910 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.167661 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:08.167910 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.167676 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:08.167910 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.167689 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:08.167910 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.167702 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:08.167910 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.167715 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:08.167910 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.167727 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:08.167910 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.167740 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:08.167910 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.167752 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:08.167910 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.167766 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:08.167910 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.167780 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:08.167910 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.167793 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:08.167910 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.167806 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:08.167910 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.167819 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:08.167910 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.167832 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl6v2\" (UniqueName: \"kubernetes.io/projected/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-kube-api-access-dl6v2\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:08.167910 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.167846 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sm9n\" (UniqueName: \"kubernetes.io/projected/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-kube-api-access-9sm9n\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:08.167910 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.167762 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istiod-ca-cert" pod "65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c" (UID: "65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:08.167910 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.167858 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:08.167910 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.167873 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bec43b9-5e3f-4c4b-b435-7926df2d53cc-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:08.167910 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.167886 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fa14f5-bd49-49aa-88ef-fd812f99e7bc-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:08.169143 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.168745 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istio-envoy" pod "65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c" (UID: "65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:08.169143 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.169111 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-kube-api-access-chgjx" pod "65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c" (UID: "65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c"). InnerVolumeSpecName "kube-api-access-chgjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:08.169541 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.169519 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istio-token" pod "65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c" (UID: "65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:08.169695 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.169680 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istio-podinfo" pod "65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c" (UID: "65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:51:08.270047 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.270006 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chgjx\" (UniqueName: \"kubernetes.io/projected/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-kube-api-access-chgjx\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:08.270047 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.270045 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:08.270047 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.270056 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:08.270281 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.270064 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:08.270281 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.270074 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:08.682836 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.682801 4471 generic.go:356] "Generic (PLEG): container finished" podID="65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c" containerID="680fb021801780d34675eb6036f760230f7b7e1fe854273780e7d450df04379e" exitCode=0 Nov 02 17:51:08.683294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.682848 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" Nov 02 17:51:08.683294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.682885 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" event={"ID":"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c","Type":"ContainerDied","Data":"680fb021801780d34675eb6036f760230f7b7e1fe854273780e7d450df04379e"} Nov 02 17:51:08.683294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.682943 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286" event={"ID":"65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c","Type":"ContainerDied","Data":"9b2f213589a32f5db86d710e4739cdf6789cda71b7797b6f72dc3633f88a1c57"} Nov 02 17:51:08.683294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.682975 4471 scope.go:122] "RemoveContainer" containerID="680fb021801780d34675eb6036f760230f7b7e1fe854273780e7d450df04379e" Nov 02 17:51:08.684756 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.684731 4471 generic.go:356] "Generic (PLEG): container finished" podID="7bec43b9-5e3f-4c4b-b435-7926df2d53cc" containerID="e692d256494448ad494843d34287ef5a4da85965a85eb81ff10633112e8e5142" exitCode=0 Nov 02 17:51:08.685634 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.684785 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" Nov 02 17:51:08.685634 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.684792 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" event={"ID":"7bec43b9-5e3f-4c4b-b435-7926df2d53cc","Type":"ContainerDied","Data":"e692d256494448ad494843d34287ef5a4da85965a85eb81ff10633112e8e5142"} Nov 02 17:51:08.685634 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.685160 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp" event={"ID":"7bec43b9-5e3f-4c4b-b435-7926df2d53cc","Type":"ContainerDied","Data":"22961d196d4d2a55622310ec757846faf4bd7b3483128f42bf6cc2a890ff44f5"} Nov 02 17:51:08.686777 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.686753 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" Nov 02 17:51:08.686868 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.686794 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m" event={"ID":"0316e437-096a-4330-b71c-3b33b139c4d9","Type":"ContainerDied","Data":"fcf7385ea1cc7c3e8c89b9fd3a47499ea0eb8f00540957dbb1dcee70a0ae1779"} Nov 02 17:51:08.688656 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.688633 4471 generic.go:356] "Generic (PLEG): container finished" podID="74fa14f5-bd49-49aa-88ef-fd812f99e7bc" containerID="5f421836c954b905f591ab4fa3eb3f6f382bc4f91242e59962f16f1fe931cdc9" exitCode=0 Nov 02 17:51:08.688730 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.688674 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" event={"ID":"74fa14f5-bd49-49aa-88ef-fd812f99e7bc","Type":"ContainerDied","Data":"5f421836c954b905f591ab4fa3eb3f6f382bc4f91242e59962f16f1fe931cdc9"} Nov 02 17:51:08.688730 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.688715 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" event={"ID":"74fa14f5-bd49-49aa-88ef-fd812f99e7bc","Type":"ContainerDied","Data":"7f3d4b5d80c297cdb7b3ce941fa91a2b1aaec90a881a1fffb8d754a5a638160f"} Nov 02 17:51:08.688824 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.688680 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl" Nov 02 17:51:08.695826 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.695808 4471 scope.go:122] "RemoveContainer" containerID="680fb021801780d34675eb6036f760230f7b7e1fe854273780e7d450df04379e" Nov 02 17:51:08.696093 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:51:08.696070 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"680fb021801780d34675eb6036f760230f7b7e1fe854273780e7d450df04379e\": container with ID starting with 680fb021801780d34675eb6036f760230f7b7e1fe854273780e7d450df04379e not found: ID does not exist" containerID="680fb021801780d34675eb6036f760230f7b7e1fe854273780e7d450df04379e" Nov 02 17:51:08.696144 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.696095 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"680fb021801780d34675eb6036f760230f7b7e1fe854273780e7d450df04379e"} err="failed to get container status \"680fb021801780d34675eb6036f760230f7b7e1fe854273780e7d450df04379e\": rpc error: code = NotFound desc = could not find container \"680fb021801780d34675eb6036f760230f7b7e1fe854273780e7d450df04379e\": container with ID starting with 680fb021801780d34675eb6036f760230f7b7e1fe854273780e7d450df04379e not found: ID does not exist" Nov 02 17:51:08.696144 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.696113 4471 scope.go:122] "RemoveContainer" containerID="e692d256494448ad494843d34287ef5a4da85965a85eb81ff10633112e8e5142" Nov 02 17:51:08.705835 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.705819 4471 scope.go:122] "RemoveContainer" containerID="e692d256494448ad494843d34287ef5a4da85965a85eb81ff10633112e8e5142" Nov 02 17:51:08.706089 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:51:08.706067 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e692d256494448ad494843d34287ef5a4da85965a85eb81ff10633112e8e5142\": container with ID starting with e692d256494448ad494843d34287ef5a4da85965a85eb81ff10633112e8e5142 not found: ID does not exist" containerID="e692d256494448ad494843d34287ef5a4da85965a85eb81ff10633112e8e5142" Nov 02 17:51:08.706138 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.706099 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e692d256494448ad494843d34287ef5a4da85965a85eb81ff10633112e8e5142"} err="failed to get container status \"e692d256494448ad494843d34287ef5a4da85965a85eb81ff10633112e8e5142\": rpc error: code = NotFound desc = could not find container \"e692d256494448ad494843d34287ef5a4da85965a85eb81ff10633112e8e5142\": container with ID starting with e692d256494448ad494843d34287ef5a4da85965a85eb81ff10633112e8e5142 not found: ID does not exist" Nov 02 17:51:08.706138 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.706117 4471 scope.go:122] "RemoveContainer" containerID="e9e0d421d895df66f634f9cc802c2666a0495c29040e2d6f268e87b5da3d400b" Nov 02 17:51:08.716222 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.716205 4471 scope.go:122] "RemoveContainer" containerID="5f421836c954b905f591ab4fa3eb3f6f382bc4f91242e59962f16f1fe931cdc9" Nov 02 17:51:08.725638 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.725622 4471 scope.go:122] "RemoveContainer" containerID="5f421836c954b905f591ab4fa3eb3f6f382bc4f91242e59962f16f1fe931cdc9" Nov 02 17:51:08.725940 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:51:08.725913 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f421836c954b905f591ab4fa3eb3f6f382bc4f91242e59962f16f1fe931cdc9\": container with ID starting with 5f421836c954b905f591ab4fa3eb3f6f382bc4f91242e59962f16f1fe931cdc9 not found: ID does not exist" containerID="5f421836c954b905f591ab4fa3eb3f6f382bc4f91242e59962f16f1fe931cdc9" Nov 02 17:51:08.725981 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.725951 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f421836c954b905f591ab4fa3eb3f6f382bc4f91242e59962f16f1fe931cdc9"} err="failed to get container status \"5f421836c954b905f591ab4fa3eb3f6f382bc4f91242e59962f16f1fe931cdc9\": rpc error: code = NotFound desc = could not find container \"5f421836c954b905f591ab4fa3eb3f6f382bc4f91242e59962f16f1fe931cdc9\": container with ID starting with 5f421836c954b905f591ab4fa3eb3f6f382bc4f91242e59962f16f1fe931cdc9 not found: ID does not exist" Nov 02 17:51:08.747191 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.747166 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m"] Nov 02 17:51:08.762213 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.762185 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-gmf4m"] Nov 02 17:51:08.799824 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.799785 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp"] Nov 02 17:51:08.808842 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.808805 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-4tkqp"] Nov 02 17:51:08.845681 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.845646 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286"] Nov 02 17:51:08.849806 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.849777 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9m286"] Nov 02 17:51:08.882724 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.882691 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl"] Nov 02 17:51:08.888094 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:08.888064 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-67ftl"] Nov 02 17:51:09.258574 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:51:09.258535 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:51:10.241454 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.241422 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1e5fcb-5494fcc9c9-fj45h"] Nov 02 17:51:10.242168 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.242153 4471 cpu_manager.go:414] "containerMap: removing container" podUID="7bec43b9-5e3f-4c4b-b435-7926df2d53cc" containerName="istio-proxy" Nov 02 17:51:10.242215 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.242169 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bec43b9-5e3f-4c4b-b435-7926df2d53cc" containerName="istio-proxy" Nov 02 17:51:10.242215 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.242189 4471 cpu_manager.go:414] "containerMap: removing container" podUID="0316e437-096a-4330-b71c-3b33b139c4d9" containerName="istio-proxy" Nov 02 17:51:10.242215 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.242194 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0316e437-096a-4330-b71c-3b33b139c4d9" containerName="istio-proxy" Nov 02 17:51:10.242215 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.242206 4471 cpu_manager.go:414] "containerMap: removing container" podUID="85637eda-7caa-4bc0-b697-1acf7f1da917" containerName="istio-proxy" Nov 02 17:51:10.242215 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.242211 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85637eda-7caa-4bc0-b697-1acf7f1da917" containerName="istio-proxy" Nov 02 17:51:10.242480 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.242233 4471 cpu_manager.go:414] "containerMap: removing container" podUID="74fa14f5-bd49-49aa-88ef-fd812f99e7bc" containerName="istio-proxy" Nov 02 17:51:10.242480 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.242238 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74fa14f5-bd49-49aa-88ef-fd812f99e7bc" containerName="istio-proxy" Nov 02 17:51:10.242480 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.242255 4471 cpu_manager.go:414] "containerMap: removing container" podUID="60f62cc3-c3a5-4d48-995f-4844ae54e13a" containerName="istio-proxy" Nov 02 17:51:10.242480 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.242259 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60f62cc3-c3a5-4d48-995f-4844ae54e13a" containerName="istio-proxy" Nov 02 17:51:10.242480 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.242269 4471 cpu_manager.go:414] "containerMap: removing container" podUID="913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324" containerName="istio-proxy" Nov 02 17:51:10.242480 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.242273 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324" containerName="istio-proxy" Nov 02 17:51:10.242480 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.242284 4471 cpu_manager.go:414] "containerMap: removing container" podUID="65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c" containerName="istio-proxy" Nov 02 17:51:10.242480 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.242288 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c" containerName="istio-proxy" Nov 02 17:51:10.242480 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.242397 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="74fa14f5-bd49-49aa-88ef-fd812f99e7bc" containerName="istio-proxy" Nov 02 17:51:10.242480 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.242407 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="60f62cc3-c3a5-4d48-995f-4844ae54e13a" containerName="istio-proxy" Nov 02 17:51:10.242480 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.242415 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="913b1cb5-db6e-4ac7-84c1-2dc9cc0f1324" containerName="istio-proxy" Nov 02 17:51:10.242480 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.242421 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c" containerName="istio-proxy" Nov 02 17:51:10.242480 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.242427 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bec43b9-5e3f-4c4b-b435-7926df2d53cc" containerName="istio-proxy" Nov 02 17:51:10.242480 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.242435 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="0316e437-096a-4330-b71c-3b33b139c4d9" containerName="istio-proxy" Nov 02 17:51:10.242480 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.242443 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="85637eda-7caa-4bc0-b697-1acf7f1da917" containerName="istio-proxy" Nov 02 17:51:10.246414 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.246396 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5fcb-5494fcc9c9-fj45h" Nov 02 17:51:10.271339 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.271297 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1e5fcb-5494fcc9c9-fj45h"] Nov 02 17:51:10.278319 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.278284 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-1e5fcb\"" Nov 02 17:51:10.278474 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.278285 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-1e5fcb-dockercfg-gzfbj\"" Nov 02 17:51:10.311529 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.311494 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b70c76d6-8ddc-4fc1-85f5-284247309434-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1e5fcb-5494fcc9c9-fj45h\" (UID: \"b70c76d6-8ddc-4fc1-85f5-284247309434\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5fcb-5494fcc9c9-fj45h" Nov 02 17:51:10.311692 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.311623 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68rwf\" (UniqueName: \"kubernetes.io/projected/b70c76d6-8ddc-4fc1-85f5-284247309434-kube-api-access-68rwf\") pod \"payload-pre-processing-e2e-api-b-1e5fcb-5494fcc9c9-fj45h\" (UID: \"b70c76d6-8ddc-4fc1-85f5-284247309434\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5fcb-5494fcc9c9-fj45h" Nov 02 17:51:10.814338 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.812536 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b70c76d6-8ddc-4fc1-85f5-284247309434-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1e5fcb-5494fcc9c9-fj45h\" (UID: \"b70c76d6-8ddc-4fc1-85f5-284247309434\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5fcb-5494fcc9c9-fj45h" Nov 02 17:51:10.817353 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.812656 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68rwf\" (UniqueName: \"kubernetes.io/projected/b70c76d6-8ddc-4fc1-85f5-284247309434-kube-api-access-68rwf\") pod \"payload-pre-processing-e2e-api-b-1e5fcb-5494fcc9c9-fj45h\" (UID: \"b70c76d6-8ddc-4fc1-85f5-284247309434\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5fcb-5494fcc9c9-fj45h" Nov 02 17:51:10.817353 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.815495 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b70c76d6-8ddc-4fc1-85f5-284247309434-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1e5fcb-5494fcc9c9-fj45h\" (UID: \"b70c76d6-8ddc-4fc1-85f5-284247309434\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5fcb-5494fcc9c9-fj45h" Nov 02 17:51:10.836133 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.836102 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0316e437-096a-4330-b71c-3b33b139c4d9" path="/var/lib/kubelet/pods/0316e437-096a-4330-b71c-3b33b139c4d9/volumes" Nov 02 17:51:10.836708 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.836688 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c" path="/var/lib/kubelet/pods/65aa2fc5-17ae-4bc5-b76a-9e1b0241ec6c/volumes" Nov 02 17:51:10.837506 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.837487 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74fa14f5-bd49-49aa-88ef-fd812f99e7bc" path="/var/lib/kubelet/pods/74fa14f5-bd49-49aa-88ef-fd812f99e7bc/volumes" Nov 02 17:51:10.838095 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.838075 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bec43b9-5e3f-4c4b-b435-7926df2d53cc" path="/var/lib/kubelet/pods/7bec43b9-5e3f-4c4b-b435-7926df2d53cc/volumes" Nov 02 17:51:10.838453 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.838434 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1e5fcb-65b654bb8f-nntvk"] Nov 02 17:51:10.840183 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.840157 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68rwf\" (UniqueName: \"kubernetes.io/projected/b70c76d6-8ddc-4fc1-85f5-284247309434-kube-api-access-68rwf\") pod \"payload-pre-processing-e2e-api-b-1e5fcb-5494fcc9c9-fj45h\" (UID: \"b70c76d6-8ddc-4fc1-85f5-284247309434\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5fcb-5494fcc9c9-fj45h" Nov 02 17:51:10.842747 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.842731 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1e5fcb-65b654bb8f-nntvk"] Nov 02 17:51:10.842819 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.842809 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1e5fcb-65b654bb8f-nntvk" Nov 02 17:51:10.860281 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:10.860262 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5fcb-5494fcc9c9-fj45h" Nov 02 17:51:11.022001 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:11.021963 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4af8a256-752b-4927-be4a-71903ce07046-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1e5fcb-65b654bb8f-nntvk\" (UID: \"4af8a256-752b-4927-be4a-71903ce07046\") " pod="openshift-ingress/payload-processing-e2e-api-b-1e5fcb-65b654bb8f-nntvk" Nov 02 17:51:11.022001 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:11.021999 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttgc5\" (UniqueName: \"kubernetes.io/projected/4af8a256-752b-4927-be4a-71903ce07046-kube-api-access-ttgc5\") pod \"payload-processing-e2e-api-b-1e5fcb-65b654bb8f-nntvk\" (UID: \"4af8a256-752b-4927-be4a-71903ce07046\") " pod="openshift-ingress/payload-processing-e2e-api-b-1e5fcb-65b654bb8f-nntvk" Nov 02 17:51:11.132152 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:11.129029 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4af8a256-752b-4927-be4a-71903ce07046-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1e5fcb-65b654bb8f-nntvk\" (UID: \"4af8a256-752b-4927-be4a-71903ce07046\") " pod="openshift-ingress/payload-processing-e2e-api-b-1e5fcb-65b654bb8f-nntvk" Nov 02 17:51:11.132152 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:11.129090 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttgc5\" (UniqueName: \"kubernetes.io/projected/4af8a256-752b-4927-be4a-71903ce07046-kube-api-access-ttgc5\") pod \"payload-processing-e2e-api-b-1e5fcb-65b654bb8f-nntvk\" (UID: \"4af8a256-752b-4927-be4a-71903ce07046\") " pod="openshift-ingress/payload-processing-e2e-api-b-1e5fcb-65b654bb8f-nntvk" Nov 02 17:51:11.147329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:11.136465 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4af8a256-752b-4927-be4a-71903ce07046-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1e5fcb-65b654bb8f-nntvk\" (UID: \"4af8a256-752b-4927-be4a-71903ce07046\") " pod="openshift-ingress/payload-processing-e2e-api-b-1e5fcb-65b654bb8f-nntvk" Nov 02 17:51:11.161648 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:11.161581 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttgc5\" (UniqueName: \"kubernetes.io/projected/4af8a256-752b-4927-be4a-71903ce07046-kube-api-access-ttgc5\") pod \"payload-processing-e2e-api-b-1e5fcb-65b654bb8f-nntvk\" (UID: \"4af8a256-752b-4927-be4a-71903ce07046\") " pod="openshift-ingress/payload-processing-e2e-api-b-1e5fcb-65b654bb8f-nntvk" Nov 02 17:51:11.166659 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:11.166627 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1e5fcb-65b654bb8f-nntvk" Nov 02 17:51:11.203390 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:51:11.203294 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb70c76d6_8ddc_4fc1_85f5_284247309434.slice/crio-6cf9498f660cf07867e67e39f380403a481c4426dce468eb7936f79ffb6630e5 WatchSource:0}: Error finding container 6cf9498f660cf07867e67e39f380403a481c4426dce468eb7936f79ffb6630e5: Status 404 returned error can't find the container with id 6cf9498f660cf07867e67e39f380403a481c4426dce468eb7936f79ffb6630e5 Nov 02 17:51:11.207105 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:11.205850 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1e5fcb-5494fcc9c9-fj45h"] Nov 02 17:51:11.475709 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:51:11.475674 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af8a256_752b_4927_be4a_71903ce07046.slice/crio-d82010b3d899254f10159e124fc3c68fd13d4e423d24c3b8b96f12f5373c7fc0 WatchSource:0}: Error finding container d82010b3d899254f10159e124fc3c68fd13d4e423d24c3b8b96f12f5373c7fc0: Status 404 returned error can't find the container with id d82010b3d899254f10159e124fc3c68fd13d4e423d24c3b8b96f12f5373c7fc0 Nov 02 17:51:11.478524 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:11.478499 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1e5fcb-65b654bb8f-nntvk"] Nov 02 17:51:11.827767 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:11.827730 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1e5fcb-65b654bb8f-nntvk" event={"ID":"4af8a256-752b-4927-be4a-71903ce07046","Type":"ContainerStarted","Data":"37b4f4c9cc5cfc242cac5b7289e8902676afb8e0c992841a9bac6b048bd4a426"} Nov 02 17:51:11.827937 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:11.827787 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1e5fcb-65b654bb8f-nntvk" event={"ID":"4af8a256-752b-4927-be4a-71903ce07046","Type":"ContainerStarted","Data":"d82010b3d899254f10159e124fc3c68fd13d4e423d24c3b8b96f12f5373c7fc0"} Nov 02 17:51:11.827937 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:11.827833 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-1e5fcb-65b654bb8f-nntvk" Nov 02 17:51:11.829423 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:11.829399 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5fcb-5494fcc9c9-fj45h" event={"ID":"b70c76d6-8ddc-4fc1-85f5-284247309434","Type":"ContainerStarted","Data":"5ab21c97c3dfa9fb2b09960190281044ea2d79b16547b100d9e6c467919f92e8"} Nov 02 17:51:11.829539 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:11.829434 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5fcb-5494fcc9c9-fj45h" event={"ID":"b70c76d6-8ddc-4fc1-85f5-284247309434","Type":"ContainerStarted","Data":"6cf9498f660cf07867e67e39f380403a481c4426dce468eb7936f79ffb6630e5"} Nov 02 17:51:11.829539 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:11.829536 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5fcb-5494fcc9c9-fj45h" Nov 02 17:51:11.873795 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:11.873744 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-1e5fcb-65b654bb8f-nntvk" podStartSLOduration=1.873728133 podStartE2EDuration="1.873728133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:11.869849299 +0000 UTC m=+1289.565204800" watchObservedRunningTime="2026-11-02 17:51:11.873728133 +0000 UTC m=+1289.569083610" Nov 02 17:51:11.913864 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:11.913822 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5fcb-5494fcc9c9-fj45h" podStartSLOduration=1.91381092 podStartE2EDuration="1.91381092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:11.912937325 +0000 UTC m=+1289.608292799" watchObservedRunningTime="2026-11-02 17:51:11.91381092 +0000 UTC m=+1289.609166383" Nov 02 17:51:12.305573 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.305539 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q"] Nov 02 17:51:12.309182 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.309162 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:51:12.328353 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.328321 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-786c42-openshift-default-dockercfg-b6zlr\"" Nov 02 17:51:12.343215 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.343174 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q"] Nov 02 17:51:12.360429 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.360398 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8b840f-677a-42c4-b132-7c99567bf1b2-istio-podinfo\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:51:12.360597 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.360574 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c8b840f-677a-42c4-b132-7c99567bf1b2-istio-token\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:51:12.360686 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.360669 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-workload-socket\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:51:12.360768 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.360755 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-istio-envoy\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:51:12.360842 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.360795 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzkms\" (UniqueName: \"kubernetes.io/projected/8c8b840f-677a-42c4-b132-7c99567bf1b2-kube-api-access-bzkms\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:51:12.360911 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.360900 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8b840f-677a-42c4-b132-7c99567bf1b2-istiod-ca-cert\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:51:12.360974 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.360958 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-workload-certs\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:51:12.361034 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.360980 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-istio-data\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:51:12.361090 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.361066 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-credential-socket\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:51:12.462515 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.462482 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-credential-socket\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:51:12.462659 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.462594 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8b840f-677a-42c4-b132-7c99567bf1b2-istio-podinfo\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:51:12.462787 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.462753 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c8b840f-677a-42c4-b132-7c99567bf1b2-istio-token\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:51:12.462951 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.462857 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-workload-socket\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:51:12.463018 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.462949 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-istio-envoy\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:51:12.463018 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.462970 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-credential-socket\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:51:12.463018 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.463002 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzkms\" (UniqueName: \"kubernetes.io/projected/8c8b840f-677a-42c4-b132-7c99567bf1b2-kube-api-access-bzkms\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:51:12.463182 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.463085 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8b840f-677a-42c4-b132-7c99567bf1b2-istiod-ca-cert\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:51:12.463182 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.463166 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-workload-certs\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:51:12.463555 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.463195 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-istio-data\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:51:12.463555 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.463268 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-workload-socket\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:51:12.463704 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.463569 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-workload-certs\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:51:12.463763 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.463749 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-istio-data\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:51:12.464361 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.464242 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8b840f-677a-42c4-b132-7c99567bf1b2-istiod-ca-cert\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:51:12.466768 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.466744 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-istio-envoy\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:51:12.468252 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.468231 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8b840f-677a-42c4-b132-7c99567bf1b2-istio-podinfo\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:51:12.468527 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.468508 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6815-a6a1c3"] Nov 02 17:51:12.472630 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.472608 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6815-a6a1c3" Nov 02 17:51:12.487590 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.487563 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzkms\" (UniqueName: \"kubernetes.io/projected/8c8b840f-677a-42c4-b132-7c99567bf1b2-kube-api-access-bzkms\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:51:12.488613 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.487875 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9tmjt\"" Nov 02 17:51:12.488822 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.488800 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c8b840f-677a-42c4-b132-7c99567bf1b2-istio-token\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:51:12.491349 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.491327 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6815-a6a1c3"] Nov 02 17:51:12.564265 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.564181 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr5cv\" (UniqueName: \"kubernetes.io/projected/97747e17-dda1-4bd2-951d-b909c32e3a6f-kube-api-access-kr5cv\") pod \"test-curl-6815-a6a1c3\" (UID: \"97747e17-dda1-4bd2-951d-b909c32e3a6f\") " pod="opendatahub/test-curl-6815-a6a1c3" Nov 02 17:51:12.623996 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.623963 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:51:12.666236 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.666203 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr5cv\" (UniqueName: \"kubernetes.io/projected/97747e17-dda1-4bd2-951d-b909c32e3a6f-kube-api-access-kr5cv\") pod \"test-curl-6815-a6a1c3\" (UID: \"97747e17-dda1-4bd2-951d-b909c32e3a6f\") " pod="opendatahub/test-curl-6815-a6a1c3" Nov 02 17:51:12.690800 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.690650 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr5cv\" (UniqueName: \"kubernetes.io/projected/97747e17-dda1-4bd2-951d-b909c32e3a6f-kube-api-access-kr5cv\") pod \"test-curl-6815-a6a1c3\" (UID: \"97747e17-dda1-4bd2-951d-b909c32e3a6f\") " pod="opendatahub/test-curl-6815-a6a1c3" Nov 02 17:51:12.821588 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:12.821515 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6815-a6a1c3" Nov 02 17:51:13.031680 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:13.030534 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:51:13.031680 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:13.030621 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:51:13.031680 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:13.030669 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:51:13.032600 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:13.032209 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q"] Nov 02 17:51:13.171045 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:13.171011 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6815-a6a1c3"] Nov 02 17:51:13.186132 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:51:13.186093 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97747e17_dda1_4bd2_951d_b909c32e3a6f.slice/crio-287df357a1cdde5fa7e1a98ab38670b4bc9c5cee5301f8fb0e74637741a6e492 WatchSource:0}: Error finding container 287df357a1cdde5fa7e1a98ab38670b4bc9c5cee5301f8fb0e74637741a6e492: Status 404 returned error can't find the container with id 287df357a1cdde5fa7e1a98ab38670b4bc9c5cee5301f8fb0e74637741a6e492 Nov 02 17:51:13.846377 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:13.846341 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6815-a6a1c3" event={"ID":"97747e17-dda1-4bd2-951d-b909c32e3a6f","Type":"ContainerStarted","Data":"7f51623cba2bd6101ab69c71cd03f4a0248da17b279e30d4a8b01c1b5c68d9a0"} Nov 02 17:51:13.846377 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:13.846387 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6815-a6a1c3" event={"ID":"97747e17-dda1-4bd2-951d-b909c32e3a6f","Type":"ContainerStarted","Data":"287df357a1cdde5fa7e1a98ab38670b4bc9c5cee5301f8fb0e74637741a6e492"} Nov 02 17:51:13.848368 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:13.848339 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" event={"ID":"8c8b840f-677a-42c4-b132-7c99567bf1b2","Type":"ContainerStarted","Data":"0fe944e83c6bcf2338b93ac65a0356312f6ed42614493eb36f55b60a9d140180"} Nov 02 17:51:13.848485 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:13.848389 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" event={"ID":"8c8b840f-677a-42c4-b132-7c99567bf1b2","Type":"ContainerStarted","Data":"45c44ddf164cb4d6cd5580cd8e0416e5a7e0a19d0a0789f9f6cf93d0b39f16a0"} Nov 02 17:51:13.886649 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:13.886600 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6815-a6a1c3" podStartSLOduration=1.8865874470000001 podStartE2EDuration="1.886587447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:13.885355775 +0000 UTC m=+1291.580711248" watchObservedRunningTime="2026-11-02 17:51:13.886587447 +0000 UTC m=+1291.581942919" Nov 02 17:51:13.943375 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:13.942334 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" podStartSLOduration=1.9422958590000001 podStartE2EDuration="1.942295859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:13.936058607 +0000 UTC m=+1291.631414104" watchObservedRunningTime="2026-11-02 17:51:13.942295859 +0000 UTC m=+1291.637651333" Nov 02 17:51:14.624394 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:14.624361 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:51:14.625519 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:14.625494 4471 patch_prober.go:29] interesting pod/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 17:51:14.625647 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:14.625544 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" podUID="8c8b840f-677a-42c4-b132-7c99567bf1b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 17:51:14.840332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:14.827219 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6815-a6a1c3"] Nov 02 17:51:14.848896 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:14.848867 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6815-a6a1c3"] Nov 02 17:51:15.624454 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:15.624423 4471 patch_prober.go:29] interesting pod/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 17:51:15.624627 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:15.624471 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" podUID="8c8b840f-677a-42c4-b132-7c99567bf1b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 17:51:15.896618 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:15.896519 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6815-a6a1c3" podUID="97747e17-dda1-4bd2-951d-b909c32e3a6f" containerName="test-curl-6815-a6a1c3" containerID="cri-o://7f51623cba2bd6101ab69c71cd03f4a0248da17b279e30d4a8b01c1b5c68d9a0" gracePeriod=2 Nov 02 17:51:15.910718 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:51:15.910693 4471 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6815-a6a1c3\" is forbidden: User \"system:node:ip-10-0-142-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-26.ec2.internal' and this object" podUID="97747e17-dda1-4bd2-951d-b909c32e3a6f" pod="opendatahub/test-curl-6815-a6a1c3" Nov 02 17:51:16.629858 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:16.626947 4471 patch_prober.go:29] interesting pod/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 17:51:16.629858 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:16.627036 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" podUID="8c8b840f-677a-42c4-b132-7c99567bf1b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 17:51:17.625343 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:17.625286 4471 patch_prober.go:29] interesting pod/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 17:51:17.625738 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:17.625363 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" podUID="8c8b840f-677a-42c4-b132-7c99567bf1b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 17:51:18.203429 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:18.203404 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6815-a6a1c3" Nov 02 17:51:18.301636 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:18.301601 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97747e17-dda1-4bd2-951d-b909c32e3a6f-kube-api-access-kr5cv\" (UniqueName: \"kubernetes.io/projected/97747e17-dda1-4bd2-951d-b909c32e3a6f-kube-api-access-kr5cv\") pod \"97747e17-dda1-4bd2-951d-b909c32e3a6f\" (UID: \"97747e17-dda1-4bd2-951d-b909c32e3a6f\") " Nov 02 17:51:18.303734 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:18.303706 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97747e17-dda1-4bd2-951d-b909c32e3a6f-kube-api-access-kr5cv" pod "97747e17-dda1-4bd2-951d-b909c32e3a6f" (UID: "97747e17-dda1-4bd2-951d-b909c32e3a6f"). InnerVolumeSpecName "kube-api-access-kr5cv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:18.346677 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:18.346637 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v"] Nov 02 17:51:18.346901 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:18.346857 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" podUID="04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd" containerName="istio-proxy" containerID="cri-o://62b0cdf9b85ee92dd073ded7ffbc21c0851e1510c9098b500896204ed1179329" gracePeriod=30 Nov 02 17:51:18.404931 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:18.404716 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr5cv\" (UniqueName: \"kubernetes.io/projected/97747e17-dda1-4bd2-951d-b909c32e3a6f-kube-api-access-kr5cv\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:18.448656 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:18.448623 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97747e17-dda1-4bd2-951d-b909c32e3a6f" path="/var/lib/kubelet/pods/97747e17-dda1-4bd2-951d-b909c32e3a6f/volumes" Nov 02 17:51:18.641329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:18.632283 4471 patch_prober.go:29] interesting pod/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 17:51:18.641329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:18.632367 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" podUID="8c8b840f-677a-42c4-b132-7c99567bf1b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 17:51:18.920171 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:18.920134 4471 generic.go:356] "Generic (PLEG): container finished" podID="97747e17-dda1-4bd2-951d-b909c32e3a6f" containerID="7f51623cba2bd6101ab69c71cd03f4a0248da17b279e30d4a8b01c1b5c68d9a0" exitCode=137 Nov 02 17:51:18.920419 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:18.920187 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6815-a6a1c3" Nov 02 17:51:18.920419 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:18.920236 4471 scope.go:122] "RemoveContainer" containerID="7f51623cba2bd6101ab69c71cd03f4a0248da17b279e30d4a8b01c1b5c68d9a0" Nov 02 17:51:18.930658 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:18.930639 4471 scope.go:122] "RemoveContainer" containerID="7f51623cba2bd6101ab69c71cd03f4a0248da17b279e30d4a8b01c1b5c68d9a0" Nov 02 17:51:18.930910 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:51:18.930891 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f51623cba2bd6101ab69c71cd03f4a0248da17b279e30d4a8b01c1b5c68d9a0\": container with ID starting with 7f51623cba2bd6101ab69c71cd03f4a0248da17b279e30d4a8b01c1b5c68d9a0 not found: ID does not exist" containerID="7f51623cba2bd6101ab69c71cd03f4a0248da17b279e30d4a8b01c1b5c68d9a0" Nov 02 17:51:18.930963 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:18.930921 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f51623cba2bd6101ab69c71cd03f4a0248da17b279e30d4a8b01c1b5c68d9a0"} err="failed to get container status \"7f51623cba2bd6101ab69c71cd03f4a0248da17b279e30d4a8b01c1b5c68d9a0\": rpc error: code = NotFound desc = could not find container \"7f51623cba2bd6101ab69c71cd03f4a0248da17b279e30d4a8b01c1b5c68d9a0\": container with ID starting with 7f51623cba2bd6101ab69c71cd03f4a0248da17b279e30d4a8b01c1b5c68d9a0 not found: ID does not exist" Nov 02 17:51:19.629905 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:19.629875 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:51:19.630267 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:19.630244 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:51:19.630956 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:19.630901 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:51:22.839539 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:22.839510 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-1e5fcb-65b654bb8f-nntvk" Nov 02 17:51:22.839967 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:22.839585 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5fcb-5494fcc9c9-fj45h" Nov 02 17:51:23.633457 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.633434 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:51:23.740447 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.740413 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-workload-certs\") pod \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " Nov 02 17:51:23.740613 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.740487 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-kube-api-access-pmvqj\" (UniqueName: \"kubernetes.io/projected/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-kube-api-access-pmvqj\") pod \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " Nov 02 17:51:23.740690 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.740649 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istiod-ca-cert\") pod \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " Nov 02 17:51:23.740751 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.740735 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-credential-socket\") pod \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " Nov 02 17:51:23.740805 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.740739 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-workload-certs" pod "04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd" (UID: "04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:23.740805 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.740764 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istio-token\" (UniqueName: \"kubernetes.io/projected/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istio-token\") pod \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " Nov 02 17:51:23.740916 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.740834 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istio-podinfo\") pod \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " Nov 02 17:51:23.740972 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.740928 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-workload-socket\") pod \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " Nov 02 17:51:23.740972 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.740960 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-credential-socket" pod "04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd" (UID: "04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:23.741077 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.740981 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istio-data\") pod \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " Nov 02 17:51:23.741077 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.741012 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istio-envoy\") pod \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\" (UID: \"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd\") " Nov 02 17:51:23.741585 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.741419 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istiod-ca-cert" pod "04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd" (UID: "04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:23.741718 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.741586 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istio-data" pod "04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd" (UID: "04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:23.741718 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.741652 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-workload-socket" pod "04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd" (UID: "04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:23.742044 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.741872 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:23.742044 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.741899 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:23.742044 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.741912 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:23.742044 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.741927 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:23.742044 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.741941 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:23.743703 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.743479 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-kube-api-access-pmvqj" pod "04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd" (UID: "04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd"). InnerVolumeSpecName "kube-api-access-pmvqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:23.743703 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.743535 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istio-podinfo" pod "04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd" (UID: "04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:51:23.743848 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.743774 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istio-token" pod "04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd" (UID: "04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:23.744106 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.744072 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istio-envoy" pod "04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd" (UID: "04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:23.843697 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.843663 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmvqj\" (UniqueName: \"kubernetes.io/projected/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-kube-api-access-pmvqj\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:23.843697 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.843693 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:23.843697 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.843703 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:23.844206 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.843711 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:23.960705 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.960629 4471 generic.go:356] "Generic (PLEG): container finished" podID="04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd" containerID="62b0cdf9b85ee92dd073ded7ffbc21c0851e1510c9098b500896204ed1179329" exitCode=0 Nov 02 17:51:23.960705 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.960680 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" Nov 02 17:51:23.960873 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.960709 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" event={"ID":"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd","Type":"ContainerDied","Data":"62b0cdf9b85ee92dd073ded7ffbc21c0851e1510c9098b500896204ed1179329"} Nov 02 17:51:23.960873 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.960764 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v" event={"ID":"04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd","Type":"ContainerDied","Data":"fe4f019f00d4a214930bf87f3ea22c26132320b8c774c58fa97b2b9f06526492"} Nov 02 17:51:23.960873 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.960785 4471 scope.go:122] "RemoveContainer" containerID="62b0cdf9b85ee92dd073ded7ffbc21c0851e1510c9098b500896204ed1179329" Nov 02 17:51:23.972882 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.972862 4471 scope.go:122] "RemoveContainer" containerID="62b0cdf9b85ee92dd073ded7ffbc21c0851e1510c9098b500896204ed1179329" Nov 02 17:51:23.973139 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:51:23.973119 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62b0cdf9b85ee92dd073ded7ffbc21c0851e1510c9098b500896204ed1179329\": container with ID starting with 62b0cdf9b85ee92dd073ded7ffbc21c0851e1510c9098b500896204ed1179329 not found: ID does not exist" containerID="62b0cdf9b85ee92dd073ded7ffbc21c0851e1510c9098b500896204ed1179329" Nov 02 17:51:23.973190 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:23.973147 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b0cdf9b85ee92dd073ded7ffbc21c0851e1510c9098b500896204ed1179329"} err="failed to get container status \"62b0cdf9b85ee92dd073ded7ffbc21c0851e1510c9098b500896204ed1179329\": rpc error: code = NotFound desc = could not find container \"62b0cdf9b85ee92dd073ded7ffbc21c0851e1510c9098b500896204ed1179329\": container with ID starting with 62b0cdf9b85ee92dd073ded7ffbc21c0851e1510c9098b500896204ed1179329 not found: ID does not exist" Nov 02 17:51:24.021802 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:24.021772 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v"] Nov 02 17:51:24.033865 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:24.033834 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-8xj7v"] Nov 02 17:51:24.442728 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:24.442696 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd" path="/var/lib/kubelet/pods/04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd/volumes" Nov 02 17:51:25.921389 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:25.918875 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-b89px"] Nov 02 17:51:27.922789 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:27.922747 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb"] Nov 02 17:51:27.924068 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:27.924037 4471 cpu_manager.go:414] "containerMap: removing container" podUID="97747e17-dda1-4bd2-951d-b909c32e3a6f" containerName="test-curl-6815-a6a1c3" Nov 02 17:51:27.924068 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:27.924067 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97747e17-dda1-4bd2-951d-b909c32e3a6f" containerName="test-curl-6815-a6a1c3" Nov 02 17:51:27.924251 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:27.924116 4471 cpu_manager.go:414] "containerMap: removing container" podUID="04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd" containerName="istio-proxy" Nov 02 17:51:27.924251 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:27.924125 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd" containerName="istio-proxy" Nov 02 17:51:27.924432 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:27.924350 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="97747e17-dda1-4bd2-951d-b909c32e3a6f" containerName="test-curl-6815-a6a1c3" Nov 02 17:51:27.924432 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:27.924375 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="04acf0e6-3d2d-41c1-a8dd-3010b57fe4cd" containerName="istio-proxy" Nov 02 17:51:27.928858 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:27.928833 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:51:27.957978 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:27.957952 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-063f48-openshift-default-dockercfg-n95mk\"" Nov 02 17:51:27.972027 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:27.971992 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb"] Nov 02 17:51:28.088350 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:28.083488 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-istio-data\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:51:28.088350 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:28.083550 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-workload-certs\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:51:28.088350 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:28.083599 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b9ada91-7257-4ad9-952a-647e1163267d-istiod-ca-cert\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:51:28.088350 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:28.083920 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxjg4\" (UniqueName: \"kubernetes.io/projected/2b9ada91-7257-4ad9-952a-647e1163267d-kube-api-access-cxjg4\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:51:28.088350 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:28.083955 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-credential-socket\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:51:28.088350 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:28.084012 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-istio-envoy\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:51:28.088350 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:28.084074 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b9ada91-7257-4ad9-952a-647e1163267d-istio-token\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:51:28.088350 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:28.084110 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b9ada91-7257-4ad9-952a-647e1163267d-istio-podinfo\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:51:28.088350 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:28.084147 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-workload-socket\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:51:28.247414 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:28.235909 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxjg4\" (UniqueName: \"kubernetes.io/projected/2b9ada91-7257-4ad9-952a-647e1163267d-kube-api-access-cxjg4\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:51:28.248007 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:28.247971 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-credential-socket\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:51:28.248209 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:28.248194 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-istio-envoy\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:51:28.248382 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:28.248362 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b9ada91-7257-4ad9-952a-647e1163267d-istio-token\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:51:28.248545 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:28.248529 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b9ada91-7257-4ad9-952a-647e1163267d-istio-podinfo\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:51:28.248668 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:28.248654 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-workload-socket\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:51:28.253487 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:28.248814 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-istio-data\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:51:28.253713 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:28.253687 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-workload-certs\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:51:28.256048 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:28.255948 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-credential-socket\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:51:28.256736 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:28.256709 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-workload-certs\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:51:28.258749 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:28.256821 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-workload-socket\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:51:28.258937 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:28.257069 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-istio-data\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:51:28.267635 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:28.267564 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-istio-envoy\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:51:28.268340 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:28.268253 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b9ada91-7257-4ad9-952a-647e1163267d-istio-podinfo\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:51:28.271230 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:28.271168 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxjg4\" (UniqueName: \"kubernetes.io/projected/2b9ada91-7257-4ad9-952a-647e1163267d-kube-api-access-cxjg4\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:51:28.382150 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:28.381523 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b9ada91-7257-4ad9-952a-647e1163267d-istiod-ca-cert\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:51:28.398083 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:28.383783 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b9ada91-7257-4ad9-952a-647e1163267d-istiod-ca-cert\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:51:28.425267 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:28.397059 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b9ada91-7257-4ad9-952a-647e1163267d-istio-token\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:51:28.567081 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:28.566810 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:51:29.003070 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:51:29.003038 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b9ada91_7257_4ad9_952a_647e1163267d.slice/crio-cfb0270677fc126c46121e474a4004e5d3b456fb22c7caec13d48b643bad7133 WatchSource:0}: Error finding container cfb0270677fc126c46121e474a4004e5d3b456fb22c7caec13d48b643bad7133: Status 404 returned error can't find the container with id cfb0270677fc126c46121e474a4004e5d3b456fb22c7caec13d48b643bad7133 Nov 02 17:51:29.004773 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:29.004744 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb"] Nov 02 17:51:29.006228 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:29.006196 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:51:29.006395 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:29.006295 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:51:29.006395 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:29.006366 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:51:30.015990 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:30.015952 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" event={"ID":"2b9ada91-7257-4ad9-952a-647e1163267d","Type":"ContainerStarted","Data":"6fbd95169d6bc1ab18fc0aa3ab752a9de8a50bb7074c63fff286b4750c4bf52a"} Nov 02 17:51:30.016444 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:30.016008 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" event={"ID":"2b9ada91-7257-4ad9-952a-647e1163267d","Type":"ContainerStarted","Data":"cfb0270677fc126c46121e474a4004e5d3b456fb22c7caec13d48b643bad7133"} Nov 02 17:51:30.079501 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:30.079447 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" podStartSLOduration=3.079431699 podStartE2EDuration="3.079431699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:30.07523102 +0000 UTC m=+1307.770586515" watchObservedRunningTime="2026-11-02 17:51:30.079431699 +0000 UTC m=+1307.774787171" Nov 02 17:51:30.572136 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:30.572102 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:51:30.578180 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:30.578152 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:51:30.920011 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:30.919924 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-b89px" podUID="7e5a459e-c932-4268-9db0-5c5b007e7a72" containerName="payload-processing" containerID="cri-o://09e94c315306c43e0ca811c6bae9dc775f25ef1107771c273a99e54d2be61633" gracePeriod=25 Nov 02 17:51:31.023351 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:31.023321 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:51:31.024378 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:31.024350 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:51:34.462695 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:34.462663 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-063f48-578558f8d7-r97k5"] Nov 02 17:51:34.466174 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:34.466156 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-063f48-578558f8d7-r97k5" Nov 02 17:51:34.486192 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:34.486164 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-063f48-578558f8d7-r97k5"] Nov 02 17:51:34.494409 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:34.494382 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-063f48-dockercfg-m7xwx\"" Nov 02 17:51:34.591652 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:34.572706 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-063f48-578558f8d7-r97k5\" (UID: \"1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-063f48-578558f8d7-r97k5" Nov 02 17:51:34.591652 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:34.572780 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwjb6\" (UniqueName: \"kubernetes.io/projected/1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9-kube-api-access-mwjb6\") pod \"payload-pre-processing-e2e-inf-a-063f48-578558f8d7-r97k5\" (UID: \"1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-063f48-578558f8d7-r97k5" Nov 02 17:51:34.591652 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:34.579278 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-063f48\"" Nov 02 17:51:34.832895 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:34.832856 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-063f48-578558f8d7-r97k5\" (UID: \"1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-063f48-578558f8d7-r97k5" Nov 02 17:51:34.834906 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:34.834873 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwjb6\" (UniqueName: \"kubernetes.io/projected/1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9-kube-api-access-mwjb6\") pod \"payload-pre-processing-e2e-inf-a-063f48-578558f8d7-r97k5\" (UID: \"1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-063f48-578558f8d7-r97k5" Nov 02 17:51:34.847709 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:34.847677 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-063f48-578558f8d7-r97k5\" (UID: \"1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-063f48-578558f8d7-r97k5" Nov 02 17:51:34.964869 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:34.959648 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwjb6\" (UniqueName: \"kubernetes.io/projected/1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9-kube-api-access-mwjb6\") pod \"payload-pre-processing-e2e-inf-a-063f48-578558f8d7-r97k5\" (UID: \"1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-063f48-578558f8d7-r97k5" Nov 02 17:51:35.080843 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:35.080802 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-063f48-578558f8d7-r97k5" Nov 02 17:51:35.837436 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:51:35.837403 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c55e1d7_ad0e_4491_a301_7d5ccf4cbfa9.slice/crio-d0cc07e363795d3a3136de1ca0cc36dc2e57d323dfb696dae419099ac51183d2 WatchSource:0}: Error finding container d0cc07e363795d3a3136de1ca0cc36dc2e57d323dfb696dae419099ac51183d2: Status 404 returned error can't find the container with id d0cc07e363795d3a3136de1ca0cc36dc2e57d323dfb696dae419099ac51183d2 Nov 02 17:51:35.839926 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:35.839902 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-063f48-578558f8d7-r97k5"] Nov 02 17:51:36.499046 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:36.496928 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-063f48-578558f8d7-r97k5" event={"ID":"1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9","Type":"ContainerStarted","Data":"6eb740b19e330c59d7eca67580fdffb8026b9019b8cbb1a9b365f37b1d264cec"} Nov 02 17:51:36.499046 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:36.497007 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-063f48-578558f8d7-r97k5" event={"ID":"1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9","Type":"ContainerStarted","Data":"d0cc07e363795d3a3136de1ca0cc36dc2e57d323dfb696dae419099ac51183d2"} Nov 02 17:51:36.499046 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:36.498398 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-063f48-578558f8d7-r97k5" Nov 02 17:51:36.553393 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:36.553331 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-063f48-578558f8d7-r97k5" podStartSLOduration=2.553297829 podStartE2EDuration="2.553297829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:36.552296183 +0000 UTC m=+1314.247651657" watchObservedRunningTime="2026-11-02 17:51:36.553297829 +0000 UTC m=+1314.248653303" Nov 02 17:51:39.238202 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:51:39.238164 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:51:39.287004 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:39.286965 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-b89px" podUID="7e5a459e-c932-4268-9db0-5c5b007e7a72" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 17:51:44.978450 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:44.978412 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz"] Nov 02 17:51:44.983638 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:44.983617 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz" Nov 02 17:51:45.009406 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:45.009372 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-1e5fcb-serving-cert\"" Nov 02 17:51:45.015903 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:45.015868 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz"] Nov 02 17:51:45.034155 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:45.034121 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdng2\" (UniqueName: \"kubernetes.io/projected/a5271597-990c-4d17-9042-82a6eddada9b-kube-api-access-jdng2\") pod \"maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz\" (UID: \"a5271597-990c-4d17-9042-82a6eddada9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz" Nov 02 17:51:45.034353 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:45.034315 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5271597-990c-4d17-9042-82a6eddada9b-metrics-certs\") pod \"maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz\" (UID: \"a5271597-990c-4d17-9042-82a6eddada9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz" Nov 02 17:51:45.034689 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:45.034662 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5271597-990c-4d17-9042-82a6eddada9b-openshift-service-ca\") pod \"maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz\" (UID: \"a5271597-990c-4d17-9042-82a6eddada9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz" Nov 02 17:51:45.034766 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:45.034704 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5271597-990c-4d17-9042-82a6eddada9b-maas-api-tls\") pod \"maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz\" (UID: \"a5271597-990c-4d17-9042-82a6eddada9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz" Nov 02 17:51:45.136018 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:45.135985 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5271597-990c-4d17-9042-82a6eddada9b-openshift-service-ca\") pod \"maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz\" (UID: \"a5271597-990c-4d17-9042-82a6eddada9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz" Nov 02 17:51:45.136212 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:45.136034 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5271597-990c-4d17-9042-82a6eddada9b-maas-api-tls\") pod \"maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz\" (UID: \"a5271597-990c-4d17-9042-82a6eddada9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz" Nov 02 17:51:45.136212 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:45.136081 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdng2\" (UniqueName: \"kubernetes.io/projected/a5271597-990c-4d17-9042-82a6eddada9b-kube-api-access-jdng2\") pod \"maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz\" (UID: \"a5271597-990c-4d17-9042-82a6eddada9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz" Nov 02 17:51:45.136212 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:45.136159 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5271597-990c-4d17-9042-82a6eddada9b-metrics-certs\") pod \"maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz\" (UID: \"a5271597-990c-4d17-9042-82a6eddada9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz" Nov 02 17:51:45.136780 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:45.136754 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5271597-990c-4d17-9042-82a6eddada9b-openshift-service-ca\") pod \"maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz\" (UID: \"a5271597-990c-4d17-9042-82a6eddada9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz" Nov 02 17:51:45.138500 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:45.138472 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5271597-990c-4d17-9042-82a6eddada9b-maas-api-tls\") pod \"maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz\" (UID: \"a5271597-990c-4d17-9042-82a6eddada9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz" Nov 02 17:51:45.138611 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:45.138552 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5271597-990c-4d17-9042-82a6eddada9b-metrics-certs\") pod \"maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz\" (UID: \"a5271597-990c-4d17-9042-82a6eddada9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz" Nov 02 17:51:45.206333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:45.206029 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdng2\" (UniqueName: \"kubernetes.io/projected/a5271597-990c-4d17-9042-82a6eddada9b-kube-api-access-jdng2\") pod \"maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz\" (UID: \"a5271597-990c-4d17-9042-82a6eddada9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz" Nov 02 17:51:45.306327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:45.303772 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz" Nov 02 17:51:45.658512 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:51:45.658483 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5271597_990c_4d17_9042_82a6eddada9b.slice/crio-da1654de8e50dcd97d27da47eda9f293d2cefb59523ac21bc03aed653c0f7e13 WatchSource:0}: Error finding container da1654de8e50dcd97d27da47eda9f293d2cefb59523ac21bc03aed653c0f7e13: Status 404 returned error can't find the container with id da1654de8e50dcd97d27da47eda9f293d2cefb59523ac21bc03aed653c0f7e13 Nov 02 17:51:45.660567 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:45.660543 4471 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:51:45.661992 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:45.661969 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz"] Nov 02 17:51:45.732538 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:45.732507 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz" event={"ID":"a5271597-990c-4d17-9042-82a6eddada9b","Type":"ContainerStarted","Data":"da1654de8e50dcd97d27da47eda9f293d2cefb59523ac21bc03aed653c0f7e13"} Nov 02 17:51:46.741442 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:46.741398 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz" event={"ID":"a5271597-990c-4d17-9042-82a6eddada9b","Type":"ContainerStarted","Data":"3a30e01022351c1ab7a40f7ad5cf2fce669f8520116b9d75c980c740793ede50"} Nov 02 17:51:46.741853 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:46.741493 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz" Nov 02 17:51:46.790001 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:46.789954 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz" podStartSLOduration=2.4756703939999998 podStartE2EDuration="2.789940445s" totalImagesPullingTime="314.270051ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:46.78541107 +0000 UTC m=+1324.480766563" watchObservedRunningTime="2026-11-02 17:51:46.789940445 +0000 UTC m=+1324.485295917" Nov 02 17:51:47.643294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:47.643261 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4"] Nov 02 17:51:47.643545 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:47.643517 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" podUID="ce398a00-0c4b-45ea-a920-acb66ae85b7f" containerName="istio-proxy" containerID="cri-o://db4079d9e3d63af66da8c3b85be9318801464214ff32c4ab22c99fe95aa6b900" gracePeriod=30 Nov 02 17:51:48.515441 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:48.515410 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-063f48-578558f8d7-r97k5" Nov 02 17:51:49.285299 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:49.285260 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-b89px" podUID="7e5a459e-c932-4268-9db0-5c5b007e7a72" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 17:51:51.539519 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:51.539488 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-b89px" Nov 02 17:51:51.620395 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:51.620356 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e5a459e-c932-4268-9db0-5c5b007e7a72-kube-api-access-sh7qx\" (UniqueName: \"kubernetes.io/projected/7e5a459e-c932-4268-9db0-5c5b007e7a72-kube-api-access-sh7qx\") pod \"7e5a459e-c932-4268-9db0-5c5b007e7a72\" (UID: \"7e5a459e-c932-4268-9db0-5c5b007e7a72\") " Nov 02 17:51:51.620658 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:51.620637 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e5a459e-c932-4268-9db0-5c5b007e7a72-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e5a459e-c932-4268-9db0-5c5b007e7a72-plugins-config-volume\") pod \"7e5a459e-c932-4268-9db0-5c5b007e7a72\" (UID: \"7e5a459e-c932-4268-9db0-5c5b007e7a72\") " Nov 02 17:51:51.620995 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:51.620971 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e5a459e-c932-4268-9db0-5c5b007e7a72-plugins-config-volume" pod "7e5a459e-c932-4268-9db0-5c5b007e7a72" (UID: "7e5a459e-c932-4268-9db0-5c5b007e7a72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:51.621419 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:51.621399 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e5a459e-c932-4268-9db0-5c5b007e7a72-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:51.622470 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:51.622448 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e5a459e-c932-4268-9db0-5c5b007e7a72-kube-api-access-sh7qx" pod "7e5a459e-c932-4268-9db0-5c5b007e7a72" (UID: "7e5a459e-c932-4268-9db0-5c5b007e7a72"). InnerVolumeSpecName "kube-api-access-sh7qx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:51.722615 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:51.722585 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh7qx\" (UniqueName: \"kubernetes.io/projected/7e5a459e-c932-4268-9db0-5c5b007e7a72-kube-api-access-sh7qx\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:51.847514 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:51.847485 4471 generic.go:356] "Generic (PLEG): container finished" podID="7e5a459e-c932-4268-9db0-5c5b007e7a72" containerID="09e94c315306c43e0ca811c6bae9dc775f25ef1107771c273a99e54d2be61633" exitCode=0 Nov 02 17:51:51.847708 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:51.847544 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-b89px" Nov 02 17:51:51.847708 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:51.847568 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-b89px" event={"ID":"7e5a459e-c932-4268-9db0-5c5b007e7a72","Type":"ContainerDied","Data":"09e94c315306c43e0ca811c6bae9dc775f25ef1107771c273a99e54d2be61633"} Nov 02 17:51:51.847708 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:51.847648 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-b89px" event={"ID":"7e5a459e-c932-4268-9db0-5c5b007e7a72","Type":"ContainerDied","Data":"c7761cce0528e9e39a75d2ac9a5a0d6f9f4202904284bfc1d544bfc8dbf59c37"} Nov 02 17:51:51.847708 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:51.847662 4471 scope.go:122] "RemoveContainer" containerID="09e94c315306c43e0ca811c6bae9dc775f25ef1107771c273a99e54d2be61633" Nov 02 17:51:51.859008 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:51.858991 4471 scope.go:122] "RemoveContainer" containerID="09e94c315306c43e0ca811c6bae9dc775f25ef1107771c273a99e54d2be61633" Nov 02 17:51:51.859318 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:51:51.859285 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09e94c315306c43e0ca811c6bae9dc775f25ef1107771c273a99e54d2be61633\": container with ID starting with 09e94c315306c43e0ca811c6bae9dc775f25ef1107771c273a99e54d2be61633 not found: ID does not exist" containerID="09e94c315306c43e0ca811c6bae9dc775f25ef1107771c273a99e54d2be61633" Nov 02 17:51:51.859392 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:51.859327 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09e94c315306c43e0ca811c6bae9dc775f25ef1107771c273a99e54d2be61633"} err="failed to get container status \"09e94c315306c43e0ca811c6bae9dc775f25ef1107771c273a99e54d2be61633\": rpc error: code = NotFound desc = could not find container \"09e94c315306c43e0ca811c6bae9dc775f25ef1107771c273a99e54d2be61633\": container with ID starting with 09e94c315306c43e0ca811c6bae9dc775f25ef1107771c273a99e54d2be61633 not found: ID does not exist" Nov 02 17:51:51.909919 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:51.909886 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-b89px"] Nov 02 17:51:51.922522 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:51.922495 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-b89px"] Nov 02 17:51:52.444721 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:52.444683 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e5a459e-c932-4268-9db0-5c5b007e7a72" path="/var/lib/kubelet/pods/7e5a459e-c932-4268-9db0-5c5b007e7a72/volumes" Nov 02 17:51:52.842722 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:52.840474 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz" Nov 02 17:51:52.903774 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:52.903465 4471 generic.go:356] "Generic (PLEG): container finished" podID="ce398a00-0c4b-45ea-a920-acb66ae85b7f" containerID="db4079d9e3d63af66da8c3b85be9318801464214ff32c4ab22c99fe95aa6b900" exitCode=0 Nov 02 17:51:52.903774 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:52.903601 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" event={"ID":"ce398a00-0c4b-45ea-a920-acb66ae85b7f","Type":"ContainerDied","Data":"db4079d9e3d63af66da8c3b85be9318801464214ff32c4ab22c99fe95aa6b900"} Nov 02 17:51:53.368970 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:53.368942 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:51:53.495001 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:53.494969 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istio-podinfo\") pod \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " Nov 02 17:51:53.495201 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:53.495007 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce398a00-0c4b-45ea-a920-acb66ae85b7f-kube-api-access-6ntzb\" (UniqueName: \"kubernetes.io/projected/ce398a00-0c4b-45ea-a920-acb66ae85b7f-kube-api-access-6ntzb\") pod \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " Nov 02 17:51:53.495201 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:53.495027 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istio-token\" (UniqueName: \"kubernetes.io/projected/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istio-token\") pod \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " Nov 02 17:51:53.495201 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:53.495137 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istiod-ca-cert\") pod \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " Nov 02 17:51:53.495391 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:53.495268 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-workload-socket\") pod \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " Nov 02 17:51:53.495391 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:53.495355 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-credential-socket\") pod \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " Nov 02 17:51:53.495650 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:53.495576 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-workload-certs\") pod \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " Nov 02 17:51:53.495981 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:53.495943 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-credential-socket" pod "ce398a00-0c4b-45ea-a920-acb66ae85b7f" (UID: "ce398a00-0c4b-45ea-a920-acb66ae85b7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:53.495981 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:53.495961 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-workload-socket" pod "ce398a00-0c4b-45ea-a920-acb66ae85b7f" (UID: "ce398a00-0c4b-45ea-a920-acb66ae85b7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:53.496221 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:53.496187 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-workload-certs" pod "ce398a00-0c4b-45ea-a920-acb66ae85b7f" (UID: "ce398a00-0c4b-45ea-a920-acb66ae85b7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:53.496382 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:53.496354 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istiod-ca-cert" pod "ce398a00-0c4b-45ea-a920-acb66ae85b7f" (UID: "ce398a00-0c4b-45ea-a920-acb66ae85b7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:53.496520 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:53.496499 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istio-envoy\") pod \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " Nov 02 17:51:53.498008 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:53.497980 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istio-data\") pod \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\" (UID: \"ce398a00-0c4b-45ea-a920-acb66ae85b7f\") " Nov 02 17:51:53.498637 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:53.498610 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istio-podinfo" pod "ce398a00-0c4b-45ea-a920-acb66ae85b7f" (UID: "ce398a00-0c4b-45ea-a920-acb66ae85b7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:51:53.498637 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:53.498608 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce398a00-0c4b-45ea-a920-acb66ae85b7f-kube-api-access-6ntzb" pod "ce398a00-0c4b-45ea-a920-acb66ae85b7f" (UID: "ce398a00-0c4b-45ea-a920-acb66ae85b7f"). InnerVolumeSpecName "kube-api-access-6ntzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:53.498828 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:53.498631 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istio-token" pod "ce398a00-0c4b-45ea-a920-acb66ae85b7f" (UID: "ce398a00-0c4b-45ea-a920-acb66ae85b7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:53.498828 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:53.498712 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istio-data" pod "ce398a00-0c4b-45ea-a920-acb66ae85b7f" (UID: "ce398a00-0c4b-45ea-a920-acb66ae85b7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:53.499383 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:53.499366 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:53.499463 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:53.499389 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:53.499463 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:53.499403 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:53.499463 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:53.499417 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:53.499463 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:53.499431 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:53.499463 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:53.499443 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:53.499463 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:53.499458 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ntzb\" (UniqueName: \"kubernetes.io/projected/ce398a00-0c4b-45ea-a920-acb66ae85b7f-kube-api-access-6ntzb\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:53.499773 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:53.499474 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:53.500406 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:53.500383 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istio-envoy" pod "ce398a00-0c4b-45ea-a920-acb66ae85b7f" (UID: "ce398a00-0c4b-45ea-a920-acb66ae85b7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:53.601594 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:53.601562 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce398a00-0c4b-45ea-a920-acb66ae85b7f-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:51:53.913612 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:53.913581 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" Nov 02 17:51:53.913612 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:53.913591 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4" event={"ID":"ce398a00-0c4b-45ea-a920-acb66ae85b7f","Type":"ContainerDied","Data":"c1ac45ad6504c66f0442ccdc77f7e594533d1797d0f4164d1d60bb6a05f4afb4"} Nov 02 17:51:53.914133 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:53.913665 4471 scope.go:122] "RemoveContainer" containerID="db4079d9e3d63af66da8c3b85be9318801464214ff32c4ab22c99fe95aa6b900" Nov 02 17:51:53.984068 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:53.984039 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4"] Nov 02 17:51:53.996028 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:53.996001 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-mfff4"] Nov 02 17:51:54.446856 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:54.446814 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce398a00-0c4b-45ea-a920-acb66ae85b7f" path="/var/lib/kubelet/pods/ce398a00-0c4b-45ea-a920-acb66ae85b7f/volumes" Nov 02 17:51:58.188359 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:58.180769 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c3c112-588f684999-xbtsd"] Nov 02 17:51:58.188359 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:58.182229 4471 cpu_manager.go:414] "containerMap: removing container" podUID="ce398a00-0c4b-45ea-a920-acb66ae85b7f" containerName="istio-proxy" Nov 02 17:51:58.188359 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:58.182259 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce398a00-0c4b-45ea-a920-acb66ae85b7f" containerName="istio-proxy" Nov 02 17:51:58.188359 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:58.182287 4471 cpu_manager.go:414] "containerMap: removing container" podUID="7e5a459e-c932-4268-9db0-5c5b007e7a72" containerName="payload-processing" Nov 02 17:51:58.188359 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:58.182296 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e5a459e-c932-4268-9db0-5c5b007e7a72" containerName="payload-processing" Nov 02 17:51:58.188359 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:58.182559 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce398a00-0c4b-45ea-a920-acb66ae85b7f" containerName="istio-proxy" Nov 02 17:51:58.188359 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:58.182578 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e5a459e-c932-4268-9db0-5c5b007e7a72" containerName="payload-processing" Nov 02 17:51:58.205278 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:58.205119 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c3c112-588f684999-xbtsd"] Nov 02 17:51:58.205278 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:58.205168 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c3c112-6cd749d5cc-mx8nw"] Nov 02 17:51:58.210146 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:58.206712 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3c112-588f684999-xbtsd" Nov 02 17:51:58.211187 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:58.211079 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c3c112-6cd749d5cc-mx8nw"] Nov 02 17:51:58.211387 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:58.211166 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c3c112-6cd749d5cc-mx8nw" Nov 02 17:51:58.235814 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:58.229683 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c3c112-dockercfg-7nh2j\"" Nov 02 17:51:58.238650 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:58.238625 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c3c112\"" Nov 02 17:51:58.319974 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:58.319936 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/090b5b1f-22f7-4b8c-acf7-e0c79a2480d1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c3c112-6cd749d5cc-mx8nw\" (UID: \"090b5b1f-22f7-4b8c-acf7-e0c79a2480d1\") " pod="openshift-ingress/payload-processing-e2e-api-a-c3c112-6cd749d5cc-mx8nw" Nov 02 17:51:58.320163 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:58.320046 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsrtp\" (UniqueName: \"kubernetes.io/projected/aecb68c6-ddce-402e-9f08-00da13bd48a3-kube-api-access-jsrtp\") pod \"payload-pre-processing-e2e-api-a-c3c112-588f684999-xbtsd\" (UID: \"aecb68c6-ddce-402e-9f08-00da13bd48a3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3c112-588f684999-xbtsd" Nov 02 17:51:58.320163 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:58.320102 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aecb68c6-ddce-402e-9f08-00da13bd48a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c3c112-588f684999-xbtsd\" (UID: \"aecb68c6-ddce-402e-9f08-00da13bd48a3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3c112-588f684999-xbtsd" Nov 02 17:51:58.320825 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:58.320789 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slg67\" (UniqueName: \"kubernetes.io/projected/090b5b1f-22f7-4b8c-acf7-e0c79a2480d1-kube-api-access-slg67\") pod \"payload-processing-e2e-api-a-c3c112-6cd749d5cc-mx8nw\" (UID: \"090b5b1f-22f7-4b8c-acf7-e0c79a2480d1\") " pod="openshift-ingress/payload-processing-e2e-api-a-c3c112-6cd749d5cc-mx8nw" Nov 02 17:51:58.422180 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:58.422142 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slg67\" (UniqueName: \"kubernetes.io/projected/090b5b1f-22f7-4b8c-acf7-e0c79a2480d1-kube-api-access-slg67\") pod \"payload-processing-e2e-api-a-c3c112-6cd749d5cc-mx8nw\" (UID: \"090b5b1f-22f7-4b8c-acf7-e0c79a2480d1\") " pod="openshift-ingress/payload-processing-e2e-api-a-c3c112-6cd749d5cc-mx8nw" Nov 02 17:51:58.422417 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:58.422289 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/090b5b1f-22f7-4b8c-acf7-e0c79a2480d1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c3c112-6cd749d5cc-mx8nw\" (UID: \"090b5b1f-22f7-4b8c-acf7-e0c79a2480d1\") " pod="openshift-ingress/payload-processing-e2e-api-a-c3c112-6cd749d5cc-mx8nw" Nov 02 17:51:58.422417 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:58.422401 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsrtp\" (UniqueName: \"kubernetes.io/projected/aecb68c6-ddce-402e-9f08-00da13bd48a3-kube-api-access-jsrtp\") pod \"payload-pre-processing-e2e-api-a-c3c112-588f684999-xbtsd\" (UID: \"aecb68c6-ddce-402e-9f08-00da13bd48a3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3c112-588f684999-xbtsd" Nov 02 17:51:58.422553 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:58.422461 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aecb68c6-ddce-402e-9f08-00da13bd48a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c3c112-588f684999-xbtsd\" (UID: \"aecb68c6-ddce-402e-9f08-00da13bd48a3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3c112-588f684999-xbtsd" Nov 02 17:51:58.423155 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:58.423126 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/090b5b1f-22f7-4b8c-acf7-e0c79a2480d1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c3c112-6cd749d5cc-mx8nw\" (UID: \"090b5b1f-22f7-4b8c-acf7-e0c79a2480d1\") " pod="openshift-ingress/payload-processing-e2e-api-a-c3c112-6cd749d5cc-mx8nw" Nov 02 17:51:58.423418 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:58.423396 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aecb68c6-ddce-402e-9f08-00da13bd48a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c3c112-588f684999-xbtsd\" (UID: \"aecb68c6-ddce-402e-9f08-00da13bd48a3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3c112-588f684999-xbtsd" Nov 02 17:51:58.447595 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:58.447528 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slg67\" (UniqueName: \"kubernetes.io/projected/090b5b1f-22f7-4b8c-acf7-e0c79a2480d1-kube-api-access-slg67\") pod \"payload-processing-e2e-api-a-c3c112-6cd749d5cc-mx8nw\" (UID: \"090b5b1f-22f7-4b8c-acf7-e0c79a2480d1\") " pod="openshift-ingress/payload-processing-e2e-api-a-c3c112-6cd749d5cc-mx8nw" Nov 02 17:51:58.448298 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:58.448269 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsrtp\" (UniqueName: \"kubernetes.io/projected/aecb68c6-ddce-402e-9f08-00da13bd48a3-kube-api-access-jsrtp\") pod \"payload-pre-processing-e2e-api-a-c3c112-588f684999-xbtsd\" (UID: \"aecb68c6-ddce-402e-9f08-00da13bd48a3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3c112-588f684999-xbtsd" Nov 02 17:51:58.540794 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:58.540764 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3c112-588f684999-xbtsd" Nov 02 17:51:58.610152 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:58.610122 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c3c112-6cd749d5cc-mx8nw" Nov 02 17:51:58.846588 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:51:58.846555 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaecb68c6_ddce_402e_9f08_00da13bd48a3.slice/crio-d6f6025da3c5a2c28122212d5cd10148af4ffc0a621c37e31f01f1fa407e5e38 WatchSource:0}: Error finding container d6f6025da3c5a2c28122212d5cd10148af4ffc0a621c37e31f01f1fa407e5e38: Status 404 returned error can't find the container with id d6f6025da3c5a2c28122212d5cd10148af4ffc0a621c37e31f01f1fa407e5e38 Nov 02 17:51:58.850416 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:58.850391 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c3c112-588f684999-xbtsd"] Nov 02 17:51:58.927833 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:58.927805 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c3c112-6cd749d5cc-mx8nw"] Nov 02 17:51:59.080787 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:59.080742 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3c112-588f684999-xbtsd" event={"ID":"aecb68c6-ddce-402e-9f08-00da13bd48a3","Type":"ContainerStarted","Data":"a9ddb485cc682982e8a8833c79e7ffcb4383cacba2c090ac30efe784561f7d33"} Nov 02 17:51:59.080971 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:59.080810 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3c112-588f684999-xbtsd" event={"ID":"aecb68c6-ddce-402e-9f08-00da13bd48a3","Type":"ContainerStarted","Data":"d6f6025da3c5a2c28122212d5cd10148af4ffc0a621c37e31f01f1fa407e5e38"} Nov 02 17:51:59.080971 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:59.080839 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3c112-588f684999-xbtsd" Nov 02 17:51:59.082510 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:59.082484 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c3c112-6cd749d5cc-mx8nw" event={"ID":"090b5b1f-22f7-4b8c-acf7-e0c79a2480d1","Type":"ContainerStarted","Data":"a30c47a6ceea5022dda47e137426323f7bad62d3cf7eb8df49e657b1df6c6620"} Nov 02 17:51:59.082608 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:59.082529 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c3c112-6cd749d5cc-mx8nw" event={"ID":"090b5b1f-22f7-4b8c-acf7-e0c79a2480d1","Type":"ContainerStarted","Data":"6117d837810ad290aa4997bbe4c6958f4ed2ce3c6d0b33253f9707a0ad79586e"} Nov 02 17:51:59.082608 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:59.082550 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c3c112-6cd749d5cc-mx8nw" Nov 02 17:51:59.126514 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:59.126433 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3c112-588f684999-xbtsd" podStartSLOduration=2.126419779 podStartE2EDuration="2.126419779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:59.122509355 +0000 UTC m=+1336.817864828" watchObservedRunningTime="2026-11-02 17:51:59.126419779 +0000 UTC m=+1336.821775296" Nov 02 17:51:59.168953 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:51:59.168902 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c3c112-6cd749d5cc-mx8nw" podStartSLOduration=2.168888615 podStartE2EDuration="2.168888615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:59.164931034 +0000 UTC m=+1336.860286525" watchObservedRunningTime="2026-11-02 17:51:59.168888615 +0000 UTC m=+1336.864244088" Nov 02 17:52:09.187050 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:52:09.187007 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:52:10.127445 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:52:10.127413 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3c112-588f684999-xbtsd" Nov 02 17:52:10.140319 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:52:10.140283 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c3c112-6cd749d5cc-mx8nw" Nov 02 17:52:25.875329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:52:25.868102 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:52:28.390219 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:52:28.390171 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded" Nov 02 17:52:28.429330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:52:28.424399 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:52:28.631381 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:52:28.620855 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 02 17:52:39.255342 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:52:39.245239 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:52:42.105113 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:52:42.105081 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-n5wdd"] Nov 02 17:52:42.149552 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:52:42.149523 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-n5wdd"] Nov 02 17:52:42.149982 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:52:42.149916 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-n5wdd" Nov 02 17:52:42.183441 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:52:42.183405 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-04e682fe\"" Nov 02 17:52:42.200366 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:52:42.199978 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-04e682fe-dockercfg-7wjxd\"" Nov 02 17:52:42.233732 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:52:42.233691 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/451e2cd1-cdf7-49f8-86a8-dbe0e371b152-plugins-config-volume\") pod \"payload-processing-e2e-mt-04e682fe-759db489b7-n5wdd\" (UID: \"451e2cd1-cdf7-49f8-86a8-dbe0e371b152\") " pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-n5wdd" Nov 02 17:52:42.234354 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:52:42.234332 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcxhz\" (UniqueName: \"kubernetes.io/projected/451e2cd1-cdf7-49f8-86a8-dbe0e371b152-kube-api-access-wcxhz\") pod \"payload-processing-e2e-mt-04e682fe-759db489b7-n5wdd\" (UID: \"451e2cd1-cdf7-49f8-86a8-dbe0e371b152\") " pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-n5wdd" Nov 02 17:52:42.336888 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:52:42.336854 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcxhz\" (UniqueName: \"kubernetes.io/projected/451e2cd1-cdf7-49f8-86a8-dbe0e371b152-kube-api-access-wcxhz\") pod \"payload-processing-e2e-mt-04e682fe-759db489b7-n5wdd\" (UID: \"451e2cd1-cdf7-49f8-86a8-dbe0e371b152\") " pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-n5wdd" Nov 02 17:52:42.337086 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:52:42.336980 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/451e2cd1-cdf7-49f8-86a8-dbe0e371b152-plugins-config-volume\") pod \"payload-processing-e2e-mt-04e682fe-759db489b7-n5wdd\" (UID: \"451e2cd1-cdf7-49f8-86a8-dbe0e371b152\") " pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-n5wdd" Nov 02 17:52:42.338195 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:52:42.338148 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/451e2cd1-cdf7-49f8-86a8-dbe0e371b152-plugins-config-volume\") pod \"payload-processing-e2e-mt-04e682fe-759db489b7-n5wdd\" (UID: \"451e2cd1-cdf7-49f8-86a8-dbe0e371b152\") " pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-n5wdd" Nov 02 17:52:42.363470 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:52:42.363403 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcxhz\" (UniqueName: \"kubernetes.io/projected/451e2cd1-cdf7-49f8-86a8-dbe0e371b152-kube-api-access-wcxhz\") pod \"payload-processing-e2e-mt-04e682fe-759db489b7-n5wdd\" (UID: \"451e2cd1-cdf7-49f8-86a8-dbe0e371b152\") " pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-n5wdd" Nov 02 17:52:42.543331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:52:42.525033 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-04e682fe-dockercfg-7wjxd\"" Nov 02 17:52:42.543331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:52:42.531804 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-n5wdd" Nov 02 17:52:42.912263 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:52:42.912224 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod451e2cd1_cdf7_49f8_86a8_dbe0e371b152.slice/crio-9789960bd8210649b33a081f96398d6b7e3e3604d6d76202fdf1784ecda7582b WatchSource:0}: Error finding container 9789960bd8210649b33a081f96398d6b7e3e3604d6d76202fdf1784ecda7582b: Status 404 returned error can't find the container with id 9789960bd8210649b33a081f96398d6b7e3e3604d6d76202fdf1784ecda7582b Nov 02 17:52:42.914330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:52:42.914290 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-n5wdd"] Nov 02 17:52:43.205385 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:52:43.205088 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-n5wdd" event={"ID":"451e2cd1-cdf7-49f8-86a8-dbe0e371b152","Type":"ContainerStarted","Data":"9789960bd8210649b33a081f96398d6b7e3e3604d6d76202fdf1784ecda7582b"} Nov 02 17:52:44.215922 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:52:44.215883 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-n5wdd" event={"ID":"451e2cd1-cdf7-49f8-86a8-dbe0e371b152","Type":"ContainerStarted","Data":"fb82ee62a1068bfa0f5180cac8ee6efcf567c708f5a8993fc8dc00f728fe59b9"} Nov 02 17:52:44.216467 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:52:44.215944 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-n5wdd" Nov 02 17:52:44.258908 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:52:44.258856 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-n5wdd" podStartSLOduration=3.258843213 podStartE2EDuration="3.258843213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:44.257659949 +0000 UTC m=+1381.953015464" watchObservedRunningTime="2026-11-02 17:52:44.258843213 +0000 UTC m=+1381.954198686" Nov 02 17:52:55.229466 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:52:55.229383 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-n5wdd" Nov 02 17:53:09.184942 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:53:09.184897 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:53:14.566334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:14.562788 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:14.602331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:14.592398 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:14.602331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:14.592393 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:14.635336 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:14.634405 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:15.000573 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:14.997797 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded" Nov 02 17:53:15.083331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:15.051854 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:15.352125 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:15.346404 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:15.572338 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:15.570505 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:15.572940 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:15.572642 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:53:15.602400 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:15.602178 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:15.606333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:15.602274 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:15.606333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:15.604195 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:53:15.637333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:15.635715 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:15.637333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:15.636784 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:53:16.063831 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:16.063789 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 02 17:53:16.064634 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:16.064606 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:53:16.537663 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:16.537626 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": context deadline exceeded" Nov 02 17:53:16.583329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:16.573509 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:16.615348 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:16.614386 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 02 17:53:16.615348 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:16.614979 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:53:16.619337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:16.616653 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:16.636416 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:16.636378 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 02 17:53:16.870764 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:16.870683 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:53:16.872741 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:16.872717 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:53:17.031458 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:17.029404 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:53:17.200989 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:17.200741 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:53:17.923415 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:17.911940 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:17.953522 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:17.952445 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:53:17.955010 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:17.954987 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t"] Nov 02 17:53:17.955159 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:17.955140 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" podUID="1ff9e12a-8b9c-4556-ae7a-fa3011dd516c" containerName="istio-proxy" containerID="cri-o://aee5c0fe15357f50fe65ff36dc095bf45753eb353079bbf5ddc997f8ac441398" gracePeriod=30 Nov 02 17:53:21.281330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.278128 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46"] Nov 02 17:53:21.285401 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.285032 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:53:21.306728 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.306656 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46"] Nov 02 17:53:21.381097 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.381063 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c3c112-openshift-default-dockercfg-6fgt6\"" Nov 02 17:53:21.385182 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.385150 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istio-envoy\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:53:21.385350 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.385328 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-workload-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:53:21.385406 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.385376 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-credential-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:53:21.385456 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.385444 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istio-token\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:53:21.385530 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.385518 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istiod-ca-cert\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:53:21.385569 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.385546 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-workload-certs\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:53:21.385699 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.385670 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istio-data\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:53:21.385829 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.385771 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istio-podinfo\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:53:21.385997 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.385968 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl5zf\" (UniqueName: \"kubernetes.io/projected/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-kube-api-access-wl5zf\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:53:21.487639 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.487597 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istio-podinfo\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:53:21.487836 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.487764 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl5zf\" (UniqueName: \"kubernetes.io/projected/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-kube-api-access-wl5zf\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:53:21.487903 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.487842 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istio-envoy\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:53:21.488022 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.487993 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-workload-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:53:21.488112 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.488070 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-credential-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:53:21.488112 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.488105 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istio-token\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:53:21.488233 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.488207 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istiod-ca-cert\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:53:21.488298 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.488248 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-workload-certs\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:53:21.488383 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.488300 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istio-data\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:53:21.488686 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.488629 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-workload-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:53:21.488812 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.488676 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-workload-certs\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:53:21.488864 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.488834 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istio-data\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:53:21.488917 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.488897 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-credential-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:53:21.489166 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.489144 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istiod-ca-cert\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:53:21.490772 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.490748 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istio-envoy\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:53:21.490927 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.490911 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istio-podinfo\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:53:21.511646 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.511622 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istio-token\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:53:21.511749 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.511712 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl5zf\" (UniqueName: \"kubernetes.io/projected/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-kube-api-access-wl5zf\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:53:21.609033 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.608955 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:53:21.939177 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.939127 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46"] Nov 02 17:53:21.939572 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:53:21.939543 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc15b9c7c_b525_4e27_9b8c_5267adedd8b2.slice/crio-f63a8a1c1fcce8115888fc7aa832bf2ac9985472c73d764bb170707754b0e4a5 WatchSource:0}: Error finding container f63a8a1c1fcce8115888fc7aa832bf2ac9985472c73d764bb170707754b0e4a5: Status 404 returned error can't find the container with id f63a8a1c1fcce8115888fc7aa832bf2ac9985472c73d764bb170707754b0e4a5 Nov 02 17:53:21.942492 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.942452 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:53:21.942578 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.942540 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:53:21.942619 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:21.942582 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:53:22.890932 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:22.890873 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" event={"ID":"c15b9c7c-b525-4e27-9b8c-5267adedd8b2","Type":"ContainerStarted","Data":"65e4a9347ffa2b1b189b8a44f59369f6f01d2c9c35125648fbeb6dcc0a461e22"} Nov 02 17:53:22.891443 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:22.890970 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" event={"ID":"c15b9c7c-b525-4e27-9b8c-5267adedd8b2","Type":"ContainerStarted","Data":"f63a8a1c1fcce8115888fc7aa832bf2ac9985472c73d764bb170707754b0e4a5"} Nov 02 17:53:23.008812 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:23.008731 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" podStartSLOduration=2.008706162 podStartE2EDuration="2.008706162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:22.96781365 +0000 UTC m=+1420.663169125" watchObservedRunningTime="2026-11-02 17:53:23.008706162 +0000 UTC m=+1420.704061635" Nov 02 17:53:23.760682 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:23.729494 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:53:23.760682 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:23.746421 4471 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 17:53:23.760682 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:23.746479 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" podUID="c15b9c7c-b525-4e27-9b8c-5267adedd8b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 17:53:24.108177 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:24.108140 4471 generic.go:356] "Generic (PLEG): container finished" podID="1ff9e12a-8b9c-4556-ae7a-fa3011dd516c" containerID="aee5c0fe15357f50fe65ff36dc095bf45753eb353079bbf5ddc997f8ac441398" exitCode=0 Nov 02 17:53:24.110239 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:24.108899 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" event={"ID":"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c","Type":"ContainerDied","Data":"aee5c0fe15357f50fe65ff36dc095bf45753eb353079bbf5ddc997f8ac441398"} Nov 02 17:53:24.110490 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:24.110478 4471 scope.go:122] "RemoveContainer" containerID="ab033f6cbb7efeb30c7490fd232a38b8081b46fcd5aa84659a37c98af3341d44" Nov 02 17:53:24.524512 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:24.524486 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:53:24.609628 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:24.609591 4471 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 17:53:24.609805 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:24.609652 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" podUID="c15b9c7c-b525-4e27-9b8c-5267adedd8b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 17:53:24.699376 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:24.691919 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istio-envoy\") pod \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " Nov 02 17:53:24.699376 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:24.692101 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-kube-api-access-2cfpk\" (UniqueName: \"kubernetes.io/projected/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-kube-api-access-2cfpk\") pod \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " Nov 02 17:53:24.699376 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:24.692138 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istio-token\" (UniqueName: \"kubernetes.io/projected/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istio-token\") pod \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " Nov 02 17:53:24.699376 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:24.692213 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-workload-socket\") pod \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " Nov 02 17:53:24.699376 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:24.692379 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istiod-ca-cert\") pod \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " Nov 02 17:53:24.699376 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:24.692435 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-credential-socket\") pod \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " Nov 02 17:53:24.699376 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:24.692580 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-workload-certs\") pod \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " Nov 02 17:53:24.699376 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:24.692627 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istio-data\") pod \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " Nov 02 17:53:24.699376 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:24.692663 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istio-podinfo\") pod \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\" (UID: \"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c\") " Nov 02 17:53:24.699376 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:24.693916 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-workload-socket" pod "1ff9e12a-8b9c-4556-ae7a-fa3011dd516c" (UID: "1ff9e12a-8b9c-4556-ae7a-fa3011dd516c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:24.699376 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:24.697011 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-workload-certs" pod "1ff9e12a-8b9c-4556-ae7a-fa3011dd516c" (UID: "1ff9e12a-8b9c-4556-ae7a-fa3011dd516c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:24.699376 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:24.697195 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istio-data" pod "1ff9e12a-8b9c-4556-ae7a-fa3011dd516c" (UID: "1ff9e12a-8b9c-4556-ae7a-fa3011dd516c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:24.699376 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:24.697610 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-credential-socket" pod "1ff9e12a-8b9c-4556-ae7a-fa3011dd516c" (UID: "1ff9e12a-8b9c-4556-ae7a-fa3011dd516c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:24.699376 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:24.697981 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istiod-ca-cert" pod "1ff9e12a-8b9c-4556-ae7a-fa3011dd516c" (UID: "1ff9e12a-8b9c-4556-ae7a-fa3011dd516c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:24.715239 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:24.715207 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istio-envoy" pod "1ff9e12a-8b9c-4556-ae7a-fa3011dd516c" (UID: "1ff9e12a-8b9c-4556-ae7a-fa3011dd516c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:24.716814 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:24.716786 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istio-token" pod "1ff9e12a-8b9c-4556-ae7a-fa3011dd516c" (UID: "1ff9e12a-8b9c-4556-ae7a-fa3011dd516c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:24.717382 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:24.717354 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istio-podinfo" pod "1ff9e12a-8b9c-4556-ae7a-fa3011dd516c" (UID: "1ff9e12a-8b9c-4556-ae7a-fa3011dd516c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:53:24.717537 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:24.717512 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-kube-api-access-2cfpk" pod "1ff9e12a-8b9c-4556-ae7a-fa3011dd516c" (UID: "1ff9e12a-8b9c-4556-ae7a-fa3011dd516c"). InnerVolumeSpecName "kube-api-access-2cfpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:24.795493 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:24.795451 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:24.795493 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:24.795489 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:24.795493 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:24.795503 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:24.795823 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:24.795515 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:24.795823 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:24.795530 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cfpk\" (UniqueName: \"kubernetes.io/projected/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-kube-api-access-2cfpk\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:24.795823 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:24.795544 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:24.795823 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:24.795558 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:24.795823 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:24.795570 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:24.795823 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:24.795583 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:25.118771 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:25.118738 4471 generic.go:356] "Generic (PLEG): container finished" podID="2b9ada91-7257-4ad9-952a-647e1163267d" containerID="6fbd95169d6bc1ab18fc0aa3ab752a9de8a50bb7074c63fff286b4750c4bf52a" exitCode=0 Nov 02 17:53:25.119215 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:25.118823 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" event={"ID":"2b9ada91-7257-4ad9-952a-647e1163267d","Type":"ContainerDied","Data":"6fbd95169d6bc1ab18fc0aa3ab752a9de8a50bb7074c63fff286b4750c4bf52a"} Nov 02 17:53:25.119294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:25.119281 4471 scope.go:122] "RemoveContainer" containerID="6fbd95169d6bc1ab18fc0aa3ab752a9de8a50bb7074c63fff286b4750c4bf52a" Nov 02 17:53:25.120379 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:25.120364 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" Nov 02 17:53:25.120471 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:25.120419 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t" event={"ID":"1ff9e12a-8b9c-4556-ae7a-fa3011dd516c","Type":"ContainerDied","Data":"4d1317b69ce028c87cb9f8b80146e05a9eac5bc42503ac4fee06bf1ca5d27cd9"} Nov 02 17:53:25.120471 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:25.120465 4471 scope.go:122] "RemoveContainer" containerID="aee5c0fe15357f50fe65ff36dc095bf45753eb353079bbf5ddc997f8ac441398" Nov 02 17:53:25.121182 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:25.121150 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:53:25.121294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:25.121216 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:53:25.121294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:25.121248 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:53:25.225768 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:25.225740 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t"] Nov 02 17:53:25.230884 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:25.230853 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-vhp6t"] Nov 02 17:53:25.609962 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:25.609930 4471 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 17:53:25.610134 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:25.609986 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" podUID="c15b9c7c-b525-4e27-9b8c-5267adedd8b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 17:53:26.131392 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:26.131359 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" event={"ID":"2b9ada91-7257-4ad9-952a-647e1163267d","Type":"ContainerStarted","Data":"9c5a818df6feb37c1e3e12fcc91db202531ddd81ca6c4fd2a2c55d2e7c1149fd"} Nov 02 17:53:26.446434 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:26.446347 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ff9e12a-8b9c-4556-ae7a-fa3011dd516c" path="/var/lib/kubelet/pods/1ff9e12a-8b9c-4556-ae7a-fa3011dd516c/volumes" Nov 02 17:53:26.578625 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:26.578587 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:53:26.578982 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:26.578954 4471 patch_prober.go:29] interesting pod/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 17:53:26.579180 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:26.579152 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" podUID="2b9ada91-7257-4ad9-952a-647e1163267d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 17:53:26.610339 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:26.610277 4471 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 17:53:26.610510 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:26.610349 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" podUID="c15b9c7c-b525-4e27-9b8c-5267adedd8b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 17:53:27.572566 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:27.572527 4471 patch_prober.go:29] interesting pod/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 17:53:27.573133 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:27.573103 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" podUID="2b9ada91-7257-4ad9-952a-647e1163267d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 17:53:27.611711 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:27.611675 4471 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 17:53:27.612019 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:27.611986 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" podUID="c15b9c7c-b525-4e27-9b8c-5267adedd8b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 17:53:28.578794 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:28.578756 4471 patch_prober.go:29] interesting pod/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 17:53:28.579410 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:28.579380 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" podUID="2b9ada91-7257-4ad9-952a-647e1163267d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 17:53:28.609860 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:28.609824 4471 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 17:53:28.610036 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:28.609877 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" podUID="c15b9c7c-b525-4e27-9b8c-5267adedd8b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 17:53:29.579829 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:29.579791 4471 patch_prober.go:29] interesting pod/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 17:53:29.580481 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:29.580437 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" podUID="2b9ada91-7257-4ad9-952a-647e1163267d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 17:53:29.615007 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:29.614976 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:53:29.615544 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:29.615525 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:53:29.616219 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:29.616200 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:53:29.634533 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:29.634509 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-227833-55fc9bbf4c-vkgwf"] Nov 02 17:53:29.635342 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:29.635327 4471 cpu_manager.go:414] "containerMap: removing container" podUID="1ff9e12a-8b9c-4556-ae7a-fa3011dd516c" containerName="istio-proxy" Nov 02 17:53:29.635411 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:29.635342 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ff9e12a-8b9c-4556-ae7a-fa3011dd516c" containerName="istio-proxy" Nov 02 17:53:29.635411 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:29.635359 4471 cpu_manager.go:414] "containerMap: removing container" podUID="1ff9e12a-8b9c-4556-ae7a-fa3011dd516c" containerName="istio-proxy" Nov 02 17:53:29.635411 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:29.635364 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ff9e12a-8b9c-4556-ae7a-fa3011dd516c" containerName="istio-proxy" Nov 02 17:53:29.635514 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:29.635491 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ff9e12a-8b9c-4556-ae7a-fa3011dd516c" containerName="istio-proxy" Nov 02 17:53:29.639235 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:29.639220 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-227833-55fc9bbf4c-vkgwf" Nov 02 17:53:29.654705 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:29.654683 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-227833\"" Nov 02 17:53:29.654803 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:29.654736 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-227833-dockercfg-jx9h4\"" Nov 02 17:53:29.691783 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:29.691751 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-227833-55fc9bbf4c-vkgwf"] Nov 02 17:53:29.831558 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:29.797419 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-227833-584bc86fbc-j9s47"] Nov 02 17:53:29.831558 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:29.809638 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ff9e12a-8b9c-4556-ae7a-fa3011dd516c" containerName="istio-proxy" Nov 02 17:53:29.831558 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:29.814260 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9207e3dc-355e-468f-a9ba-7d4d83c1e8cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-227833-55fc9bbf4c-vkgwf\" (UID: \"9207e3dc-355e-468f-a9ba-7d4d83c1e8cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-227833-55fc9bbf4c-vkgwf" Nov 02 17:53:29.831558 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:29.814358 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4djkk\" (UniqueName: \"kubernetes.io/projected/9207e3dc-355e-468f-a9ba-7d4d83c1e8cd-kube-api-access-4djkk\") pod \"payload-pre-processing-e2e-api-b-227833-55fc9bbf4c-vkgwf\" (UID: \"9207e3dc-355e-468f-a9ba-7d4d83c1e8cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-227833-55fc9bbf4c-vkgwf" Nov 02 17:53:29.853502 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:29.847207 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-227833-584bc86fbc-j9s47" Nov 02 17:53:29.979942 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:29.979903 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9gg7\" (UniqueName: \"kubernetes.io/projected/9c40f8a7-55a8-479b-8305-68f236fd9749-kube-api-access-s9gg7\") pod \"payload-processing-e2e-api-b-227833-584bc86fbc-j9s47\" (UID: \"9c40f8a7-55a8-479b-8305-68f236fd9749\") " pod="openshift-ingress/payload-processing-e2e-api-b-227833-584bc86fbc-j9s47" Nov 02 17:53:30.313329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:30.312576 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c40f8a7-55a8-479b-8305-68f236fd9749-plugins-config-volume\") pod \"payload-processing-e2e-api-b-227833-584bc86fbc-j9s47\" (UID: \"9c40f8a7-55a8-479b-8305-68f236fd9749\") " pod="openshift-ingress/payload-processing-e2e-api-b-227833-584bc86fbc-j9s47" Nov 02 17:53:30.313329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:30.312693 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9207e3dc-355e-468f-a9ba-7d4d83c1e8cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-227833-55fc9bbf4c-vkgwf\" (UID: \"9207e3dc-355e-468f-a9ba-7d4d83c1e8cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-227833-55fc9bbf4c-vkgwf" Nov 02 17:53:30.313329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:30.312766 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4djkk\" (UniqueName: \"kubernetes.io/projected/9207e3dc-355e-468f-a9ba-7d4d83c1e8cd-kube-api-access-4djkk\") pod \"payload-pre-processing-e2e-api-b-227833-55fc9bbf4c-vkgwf\" (UID: \"9207e3dc-355e-468f-a9ba-7d4d83c1e8cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-227833-55fc9bbf4c-vkgwf" Nov 02 17:53:30.318150 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:30.318112 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9207e3dc-355e-468f-a9ba-7d4d83c1e8cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-227833-55fc9bbf4c-vkgwf\" (UID: \"9207e3dc-355e-468f-a9ba-7d4d83c1e8cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-227833-55fc9bbf4c-vkgwf" Nov 02 17:53:30.343367 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:30.342868 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-227833-584bc86fbc-j9s47"] Nov 02 17:53:30.393120 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:30.393080 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4djkk\" (UniqueName: \"kubernetes.io/projected/9207e3dc-355e-468f-a9ba-7d4d83c1e8cd-kube-api-access-4djkk\") pod \"payload-pre-processing-e2e-api-b-227833-55fc9bbf4c-vkgwf\" (UID: \"9207e3dc-355e-468f-a9ba-7d4d83c1e8cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-227833-55fc9bbf4c-vkgwf" Nov 02 17:53:30.415214 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:30.415181 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9gg7\" (UniqueName: \"kubernetes.io/projected/9c40f8a7-55a8-479b-8305-68f236fd9749-kube-api-access-s9gg7\") pod \"payload-processing-e2e-api-b-227833-584bc86fbc-j9s47\" (UID: \"9c40f8a7-55a8-479b-8305-68f236fd9749\") " pod="openshift-ingress/payload-processing-e2e-api-b-227833-584bc86fbc-j9s47" Nov 02 17:53:30.415393 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:30.415277 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c40f8a7-55a8-479b-8305-68f236fd9749-plugins-config-volume\") pod \"payload-processing-e2e-api-b-227833-584bc86fbc-j9s47\" (UID: \"9c40f8a7-55a8-479b-8305-68f236fd9749\") " pod="openshift-ingress/payload-processing-e2e-api-b-227833-584bc86fbc-j9s47" Nov 02 17:53:30.415931 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:30.415906 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c40f8a7-55a8-479b-8305-68f236fd9749-plugins-config-volume\") pod \"payload-processing-e2e-api-b-227833-584bc86fbc-j9s47\" (UID: \"9c40f8a7-55a8-479b-8305-68f236fd9749\") " pod="openshift-ingress/payload-processing-e2e-api-b-227833-584bc86fbc-j9s47" Nov 02 17:53:30.439169 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:30.439145 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9gg7\" (UniqueName: \"kubernetes.io/projected/9c40f8a7-55a8-479b-8305-68f236fd9749-kube-api-access-s9gg7\") pod \"payload-processing-e2e-api-b-227833-584bc86fbc-j9s47\" (UID: \"9c40f8a7-55a8-479b-8305-68f236fd9749\") " pod="openshift-ingress/payload-processing-e2e-api-b-227833-584bc86fbc-j9s47" Nov 02 17:53:30.569380 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:30.561560 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-227833-55fc9bbf4c-vkgwf" Nov 02 17:53:30.580997 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:30.580491 4471 patch_prober.go:29] interesting pod/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 17:53:30.580997 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:30.580548 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" podUID="2b9ada91-7257-4ad9-952a-647e1163267d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 17:53:30.644286 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:30.644257 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-227833-584bc86fbc-j9s47" Nov 02 17:53:30.891941 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:53:30.891907 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9207e3dc_355e_468f_a9ba_7d4d83c1e8cd.slice/crio-463095249fb36a5a9c57fa014aa4914689762e219b0523a7a3c0e2db72835f9a WatchSource:0}: Error finding container 463095249fb36a5a9c57fa014aa4914689762e219b0523a7a3c0e2db72835f9a: Status 404 returned error can't find the container with id 463095249fb36a5a9c57fa014aa4914689762e219b0523a7a3c0e2db72835f9a Nov 02 17:53:30.892396 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:30.892374 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-227833-55fc9bbf4c-vkgwf"] Nov 02 17:53:30.995653 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:53:30.995625 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c40f8a7_55a8_479b_8305_68f236fd9749.slice/crio-d8cc2eb756d0abfb1b4cdb6b3fa04232451327bb66178fc4d11eab66fe7dcb1b WatchSource:0}: Error finding container d8cc2eb756d0abfb1b4cdb6b3fa04232451327bb66178fc4d11eab66fe7dcb1b: Status 404 returned error can't find the container with id d8cc2eb756d0abfb1b4cdb6b3fa04232451327bb66178fc4d11eab66fe7dcb1b Nov 02 17:53:31.000515 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:31.000486 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-227833-584bc86fbc-j9s47"] Nov 02 17:53:31.023244 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:31.023221 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:53:31.425378 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:31.423940 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-227833-55fc9bbf4c-vkgwf" event={"ID":"9207e3dc-355e-468f-a9ba-7d4d83c1e8cd","Type":"ContainerStarted","Data":"aa296b89c4ce59399a1aa6a1261332052afd61138f6637bb7f743b2810faecda"} Nov 02 17:53:31.425378 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:31.424016 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-227833-55fc9bbf4c-vkgwf" event={"ID":"9207e3dc-355e-468f-a9ba-7d4d83c1e8cd","Type":"ContainerStarted","Data":"463095249fb36a5a9c57fa014aa4914689762e219b0523a7a3c0e2db72835f9a"} Nov 02 17:53:31.425718 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:31.425344 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-227833-55fc9bbf4c-vkgwf" Nov 02 17:53:31.429481 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:31.429383 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-227833-584bc86fbc-j9s47" event={"ID":"9c40f8a7-55a8-479b-8305-68f236fd9749","Type":"ContainerStarted","Data":"7086d0eca5abcc4af9a630c6123a1bb4e8ef8a986d9629fa49d9a71d0b0ab48a"} Nov 02 17:53:31.429481 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:31.429445 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-227833-584bc86fbc-j9s47" event={"ID":"9c40f8a7-55a8-479b-8305-68f236fd9749","Type":"ContainerStarted","Data":"d8cc2eb756d0abfb1b4cdb6b3fa04232451327bb66178fc4d11eab66fe7dcb1b"} Nov 02 17:53:31.430005 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:31.429952 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-227833-584bc86fbc-j9s47" Nov 02 17:53:31.473340 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:31.472545 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-227833-55fc9bbf4c-vkgwf" podStartSLOduration=2.472524265 podStartE2EDuration="2.472524265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:31.469782245 +0000 UTC m=+1429.165137781" watchObservedRunningTime="2026-11-02 17:53:31.472524265 +0000 UTC m=+1429.167879738" Nov 02 17:53:31.545352 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:31.541422 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-227833-584bc86fbc-j9s47" podStartSLOduration=2.541401132 podStartE2EDuration="2.541401132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:31.528929861 +0000 UTC m=+1429.224285338" watchObservedRunningTime="2026-11-02 17:53:31.541401132 +0000 UTC m=+1429.236756606" Nov 02 17:53:31.575101 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:31.574664 4471 patch_prober.go:29] interesting pod/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 17:53:31.575101 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:31.574726 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" podUID="2b9ada91-7257-4ad9-952a-647e1163267d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 17:53:32.577185 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:32.577148 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:53:32.578332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:32.578278 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 17:53:37.699113 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:37.699079 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn"] Nov 02 17:53:37.699534 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:37.699300 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" podUID="aee26f3f-b249-457f-a1e8-77791c926ed8" containerName="istio-proxy" containerID="cri-o://622ce3377c17fef55523b5917a0ef8bd7e92084f8d70a91add39fcccf771b01a" gracePeriod=30 Nov 02 17:53:37.715282 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:37.715247 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4"] Nov 02 17:53:37.715589 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:37.715552 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" podUID="b82b62a4-8f2a-498a-99f0-9dea59b90b07" containerName="istio-proxy" containerID="cri-o://18d262b8980d88082cadbee27fa5b69bf56ee167c1900c59877069e6b06f8e6a" gracePeriod=30 Nov 02 17:53:38.905328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:38.904918 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:53:39.201069 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:53:39.200990 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:53:42.594494 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:42.594457 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-227833-584bc86fbc-j9s47" Nov 02 17:53:43.017690 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.017655 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:53:43.064883 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.064847 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:53:43.097974 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.097938 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-workload-socket\") pod \"aee26f3f-b249-457f-a1e8-77791c926ed8\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " Nov 02 17:53:43.098153 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.098007 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aee26f3f-b249-457f-a1e8-77791c926ed8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aee26f3f-b249-457f-a1e8-77791c926ed8-istiod-ca-cert\") pod \"aee26f3f-b249-457f-a1e8-77791c926ed8\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " Nov 02 17:53:43.098153 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.098071 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-credential-socket\") pod \"aee26f3f-b249-457f-a1e8-77791c926ed8\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " Nov 02 17:53:43.098153 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.098129 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aee26f3f-b249-457f-a1e8-77791c926ed8-istio-token\" (UniqueName: \"kubernetes.io/projected/aee26f3f-b249-457f-a1e8-77791c926ed8-istio-token\") pod \"aee26f3f-b249-457f-a1e8-77791c926ed8\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " Nov 02 17:53:43.098267 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.098223 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-istio-envoy\") pod \"aee26f3f-b249-457f-a1e8-77791c926ed8\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " Nov 02 17:53:43.098321 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.098242 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-workload-socket" pod "aee26f3f-b249-457f-a1e8-77791c926ed8" (UID: "aee26f3f-b249-457f-a1e8-77791c926ed8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:43.098321 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.098271 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-workload-certs\") pod \"aee26f3f-b249-457f-a1e8-77791c926ed8\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " Nov 02 17:53:43.098478 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.098457 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aee26f3f-b249-457f-a1e8-77791c926ed8-istiod-ca-cert" pod "aee26f3f-b249-457f-a1e8-77791c926ed8" (UID: "aee26f3f-b249-457f-a1e8-77791c926ed8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:43.098585 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.098559 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-workload-certs" pod "aee26f3f-b249-457f-a1e8-77791c926ed8" (UID: "aee26f3f-b249-457f-a1e8-77791c926ed8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:43.098682 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.098666 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aee26f3f-b249-457f-a1e8-77791c926ed8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aee26f3f-b249-457f-a1e8-77791c926ed8-istio-podinfo\") pod \"aee26f3f-b249-457f-a1e8-77791c926ed8\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " Nov 02 17:53:43.098756 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.098741 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-istio-data\") pod \"aee26f3f-b249-457f-a1e8-77791c926ed8\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " Nov 02 17:53:43.098826 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.098810 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aee26f3f-b249-457f-a1e8-77791c926ed8-kube-api-access-7tpxv\" (UniqueName: \"kubernetes.io/projected/aee26f3f-b249-457f-a1e8-77791c926ed8-kube-api-access-7tpxv\") pod \"aee26f3f-b249-457f-a1e8-77791c926ed8\" (UID: \"aee26f3f-b249-457f-a1e8-77791c926ed8\") " Nov 02 17:53:43.099127 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.099100 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-istio-data" pod "aee26f3f-b249-457f-a1e8-77791c926ed8" (UID: "aee26f3f-b249-457f-a1e8-77791c926ed8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:43.099683 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.099657 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-credential-socket" pod "aee26f3f-b249-457f-a1e8-77791c926ed8" (UID: "aee26f3f-b249-457f-a1e8-77791c926ed8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:43.099829 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.099794 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:43.099829 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.099821 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:43.099973 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.099837 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aee26f3f-b249-457f-a1e8-77791c926ed8-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:43.099973 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.099852 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:43.099973 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.099866 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:43.101246 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.101222 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aee26f3f-b249-457f-a1e8-77791c926ed8-istio-podinfo" pod "aee26f3f-b249-457f-a1e8-77791c926ed8" (UID: "aee26f3f-b249-457f-a1e8-77791c926ed8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:53:43.101410 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.101255 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-istio-envoy" pod "aee26f3f-b249-457f-a1e8-77791c926ed8" (UID: "aee26f3f-b249-457f-a1e8-77791c926ed8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:43.102033 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.102014 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee26f3f-b249-457f-a1e8-77791c926ed8-kube-api-access-7tpxv" pod "aee26f3f-b249-457f-a1e8-77791c926ed8" (UID: "aee26f3f-b249-457f-a1e8-77791c926ed8"). InnerVolumeSpecName "kube-api-access-7tpxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:43.102127 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.102041 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee26f3f-b249-457f-a1e8-77791c926ed8-istio-token" pod "aee26f3f-b249-457f-a1e8-77791c926ed8" (UID: "aee26f3f-b249-457f-a1e8-77791c926ed8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:43.200522 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.200445 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-credential-socket\") pod \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " Nov 02 17:53:43.200666 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.200544 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b82b62a4-8f2a-498a-99f0-9dea59b90b07-kube-api-access-4rxpf\" (UniqueName: \"kubernetes.io/projected/b82b62a4-8f2a-498a-99f0-9dea59b90b07-kube-api-access-4rxpf\") pod \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " Nov 02 17:53:43.200666 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.200600 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-workload-socket\") pod \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " Nov 02 17:53:43.200666 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.200661 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istio-envoy\") pod \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " Nov 02 17:53:43.200768 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.200709 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-credential-socket" pod "b82b62a4-8f2a-498a-99f0-9dea59b90b07" (UID: "b82b62a4-8f2a-498a-99f0-9dea59b90b07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:43.200768 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.200762 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istio-podinfo\") pod \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " Nov 02 17:53:43.200850 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.200811 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istio-token\" (UniqueName: \"kubernetes.io/projected/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istio-token\") pod \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " Nov 02 17:53:43.201101 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.200968 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-workload-socket" pod "b82b62a4-8f2a-498a-99f0-9dea59b90b07" (UID: "b82b62a4-8f2a-498a-99f0-9dea59b90b07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:43.201101 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.200990 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istiod-ca-cert\") pod \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " Nov 02 17:53:43.201361 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.201193 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istio-data\") pod \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " Nov 02 17:53:43.201361 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.201241 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-workload-certs\") pod \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\" (UID: \"b82b62a4-8f2a-498a-99f0-9dea59b90b07\") " Nov 02 17:53:43.201361 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.201342 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istiod-ca-cert" pod "b82b62a4-8f2a-498a-99f0-9dea59b90b07" (UID: "b82b62a4-8f2a-498a-99f0-9dea59b90b07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:43.202597 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.201620 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istio-data" pod "b82b62a4-8f2a-498a-99f0-9dea59b90b07" (UID: "b82b62a4-8f2a-498a-99f0-9dea59b90b07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:43.202597 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.201694 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-workload-certs" pod "b82b62a4-8f2a-498a-99f0-9dea59b90b07" (UID: "b82b62a4-8f2a-498a-99f0-9dea59b90b07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:43.202597 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.202180 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aee26f3f-b249-457f-a1e8-77791c926ed8-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:43.202597 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.202203 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tpxv\" (UniqueName: \"kubernetes.io/projected/aee26f3f-b249-457f-a1e8-77791c926ed8-kube-api-access-7tpxv\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:43.202597 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.202219 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:43.202597 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.202232 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aee26f3f-b249-457f-a1e8-77791c926ed8-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:43.202597 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.202245 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:43.202597 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.202258 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aee26f3f-b249-457f-a1e8-77791c926ed8-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:43.202597 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.202270 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:43.202597 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.202285 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:43.202597 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.202298 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:43.203143 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.203037 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istio-envoy" pod "b82b62a4-8f2a-498a-99f0-9dea59b90b07" (UID: "b82b62a4-8f2a-498a-99f0-9dea59b90b07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:43.203143 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.203069 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istio-podinfo" pod "b82b62a4-8f2a-498a-99f0-9dea59b90b07" (UID: "b82b62a4-8f2a-498a-99f0-9dea59b90b07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:53:43.203246 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.203230 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b82b62a4-8f2a-498a-99f0-9dea59b90b07-kube-api-access-4rxpf" pod "b82b62a4-8f2a-498a-99f0-9dea59b90b07" (UID: "b82b62a4-8f2a-498a-99f0-9dea59b90b07"). InnerVolumeSpecName "kube-api-access-4rxpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:43.203480 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.203452 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istio-token" pod "b82b62a4-8f2a-498a-99f0-9dea59b90b07" (UID: "b82b62a4-8f2a-498a-99f0-9dea59b90b07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:43.304272 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.304234 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:43.304272 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.304269 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:43.304272 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.304281 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b82b62a4-8f2a-498a-99f0-9dea59b90b07-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:43.304540 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.304293 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rxpf\" (UniqueName: \"kubernetes.io/projected/b82b62a4-8f2a-498a-99f0-9dea59b90b07-kube-api-access-4rxpf\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:43.446913 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.446878 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-227833-55fc9bbf4c-vkgwf" Nov 02 17:53:43.813338 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.808663 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b"] Nov 02 17:53:43.813338 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.810079 4471 cpu_manager.go:414] "containerMap: removing container" podUID="b82b62a4-8f2a-498a-99f0-9dea59b90b07" containerName="istio-proxy" Nov 02 17:53:43.813338 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.810104 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b82b62a4-8f2a-498a-99f0-9dea59b90b07" containerName="istio-proxy" Nov 02 17:53:43.813338 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.810172 4471 cpu_manager.go:414] "containerMap: removing container" podUID="aee26f3f-b249-457f-a1e8-77791c926ed8" containerName="istio-proxy" Nov 02 17:53:43.813338 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.810181 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aee26f3f-b249-457f-a1e8-77791c926ed8" containerName="istio-proxy" Nov 02 17:53:43.813338 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.810397 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="aee26f3f-b249-457f-a1e8-77791c926ed8" containerName="istio-proxy" Nov 02 17:53:43.813338 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.810421 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="b82b62a4-8f2a-498a-99f0-9dea59b90b07" containerName="istio-proxy" Nov 02 17:53:43.854409 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.843428 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:53:43.854409 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.853971 4471 generic.go:356] "Generic (PLEG): container finished" podID="aee26f3f-b249-457f-a1e8-77791c926ed8" containerID="622ce3377c17fef55523b5917a0ef8bd7e92084f8d70a91add39fcccf771b01a" exitCode=0 Nov 02 17:53:43.854409 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.854301 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" event={"ID":"aee26f3f-b249-457f-a1e8-77791c926ed8","Type":"ContainerDied","Data":"622ce3377c17fef55523b5917a0ef8bd7e92084f8d70a91add39fcccf771b01a"} Nov 02 17:53:43.854409 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.854383 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" event={"ID":"aee26f3f-b249-457f-a1e8-77791c926ed8","Type":"ContainerDied","Data":"97e87ea0565c73c4e95e4c506f9a0b6dc0975a3447a66a34c44f4b98b48cd6b4"} Nov 02 17:53:43.854744 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.854437 4471 scope.go:122] "RemoveContainer" containerID="622ce3377c17fef55523b5917a0ef8bd7e92084f8d70a91add39fcccf771b01a" Nov 02 17:53:43.854744 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.854661 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn" Nov 02 17:53:43.885366 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.880908 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2f6787-openshift-default-dockercfg-5vbm5\"" Nov 02 17:53:44.047455 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.984666 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/341883ea-c8c4-49dd-ba87-ef0a3a539065-istio-token\") pod \"e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:53:44.047455 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.984775 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-istio-envoy\") pod \"e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:53:44.047455 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.984813 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-workload-certs\") pod \"e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:53:44.047455 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.984881 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-credential-socket\") pod \"e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:53:44.047455 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.984928 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/341883ea-c8c4-49dd-ba87-ef0a3a539065-istio-podinfo\") pod \"e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:53:44.047455 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.984989 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-istio-data\") pod \"e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:53:44.047455 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.985040 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-229bb\" (UniqueName: \"kubernetes.io/projected/341883ea-c8c4-49dd-ba87-ef0a3a539065-kube-api-access-229bb\") pod \"e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:53:44.047455 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.985122 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/341883ea-c8c4-49dd-ba87-ef0a3a539065-istiod-ca-cert\") pod \"e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:53:44.047455 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:43.985185 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-workload-socket\") pod \"e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:53:44.085658 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.057541 4471 generic.go:356] "Generic (PLEG): container finished" podID="b82b62a4-8f2a-498a-99f0-9dea59b90b07" containerID="18d262b8980d88082cadbee27fa5b69bf56ee167c1900c59877069e6b06f8e6a" exitCode=0 Nov 02 17:53:44.085658 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.085322 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" event={"ID":"b82b62a4-8f2a-498a-99f0-9dea59b90b07","Type":"ContainerDied","Data":"18d262b8980d88082cadbee27fa5b69bf56ee167c1900c59877069e6b06f8e6a"} Nov 02 17:53:44.085658 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.085484 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" event={"ID":"b82b62a4-8f2a-498a-99f0-9dea59b90b07","Type":"ContainerDied","Data":"761ec5437454c9c36b92b1280dc2f7f598f441d014983087d86511c9a9f1fa3b"} Nov 02 17:53:44.086225 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.086201 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4" Nov 02 17:53:44.199074 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.199036 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b"] Nov 02 17:53:44.270396 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.269063 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-istio-data\") pod \"e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:53:44.270396 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.269180 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-229bb\" (UniqueName: \"kubernetes.io/projected/341883ea-c8c4-49dd-ba87-ef0a3a539065-kube-api-access-229bb\") pod \"e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:53:44.270396 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.269294 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/341883ea-c8c4-49dd-ba87-ef0a3a539065-istiod-ca-cert\") pod \"e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:53:44.270396 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.269385 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-workload-socket\") pod \"e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:53:44.270396 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.269617 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/341883ea-c8c4-49dd-ba87-ef0a3a539065-istio-token\") pod \"e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:53:44.270396 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.269712 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-istio-envoy\") pod \"e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:53:44.270396 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.269753 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-workload-certs\") pod \"e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:53:44.270396 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.269830 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-credential-socket\") pod \"e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:53:44.270396 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.269882 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/341883ea-c8c4-49dd-ba87-ef0a3a539065-istio-podinfo\") pod \"e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:53:44.271488 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.271011 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-workload-socket\") pod \"e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:53:44.271488 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.271350 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-istio-data\") pod \"e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:53:44.280499 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.273089 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/341883ea-c8c4-49dd-ba87-ef0a3a539065-istiod-ca-cert\") pod \"e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:53:44.280499 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.273673 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-workload-certs\") pod \"e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:53:44.280499 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.273930 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-credential-socket\") pod \"e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:53:44.280499 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.278714 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn"] Nov 02 17:53:44.326324 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.326230 4471 scope.go:122] "RemoveContainer" containerID="622ce3377c17fef55523b5917a0ef8bd7e92084f8d70a91add39fcccf771b01a" Nov 02 17:53:44.332437 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.332040 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/341883ea-c8c4-49dd-ba87-ef0a3a539065-istio-podinfo\") pod \"e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:53:44.332891 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:53:44.332858 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"622ce3377c17fef55523b5917a0ef8bd7e92084f8d70a91add39fcccf771b01a\": container with ID starting with 622ce3377c17fef55523b5917a0ef8bd7e92084f8d70a91add39fcccf771b01a not found: ID does not exist" containerID="622ce3377c17fef55523b5917a0ef8bd7e92084f8d70a91add39fcccf771b01a" Nov 02 17:53:44.332993 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.332904 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"622ce3377c17fef55523b5917a0ef8bd7e92084f8d70a91add39fcccf771b01a"} err="failed to get container status \"622ce3377c17fef55523b5917a0ef8bd7e92084f8d70a91add39fcccf771b01a\": rpc error: code = NotFound desc = could not find container \"622ce3377c17fef55523b5917a0ef8bd7e92084f8d70a91add39fcccf771b01a\": container with ID starting with 622ce3377c17fef55523b5917a0ef8bd7e92084f8d70a91add39fcccf771b01a not found: ID does not exist" Nov 02 17:53:44.332993 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.332933 4471 scope.go:122] "RemoveContainer" containerID="18d262b8980d88082cadbee27fa5b69bf56ee167c1900c59877069e6b06f8e6a" Nov 02 17:53:44.333574 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.333553 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-istio-envoy\") pod \"e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:53:44.334080 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.334062 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-bvbvn"] Nov 02 17:53:44.337542 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.337516 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/341883ea-c8c4-49dd-ba87-ef0a3a539065-istio-token\") pod \"e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:53:44.337946 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.337921 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-229bb\" (UniqueName: \"kubernetes.io/projected/341883ea-c8c4-49dd-ba87-ef0a3a539065-kube-api-access-229bb\") pod \"e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:53:44.344843 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.344778 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4"] Nov 02 17:53:44.349649 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.349628 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-jvjv4"] Nov 02 17:53:44.373125 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.373104 4471 scope.go:122] "RemoveContainer" containerID="18d262b8980d88082cadbee27fa5b69bf56ee167c1900c59877069e6b06f8e6a" Nov 02 17:53:44.373399 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:53:44.373377 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18d262b8980d88082cadbee27fa5b69bf56ee167c1900c59877069e6b06f8e6a\": container with ID starting with 18d262b8980d88082cadbee27fa5b69bf56ee167c1900c59877069e6b06f8e6a not found: ID does not exist" containerID="18d262b8980d88082cadbee27fa5b69bf56ee167c1900c59877069e6b06f8e6a" Nov 02 17:53:44.373462 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.373412 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18d262b8980d88082cadbee27fa5b69bf56ee167c1900c59877069e6b06f8e6a"} err="failed to get container status \"18d262b8980d88082cadbee27fa5b69bf56ee167c1900c59877069e6b06f8e6a\": rpc error: code = NotFound desc = could not find container \"18d262b8980d88082cadbee27fa5b69bf56ee167c1900c59877069e6b06f8e6a\": container with ID starting with 18d262b8980d88082cadbee27fa5b69bf56ee167c1900c59877069e6b06f8e6a not found: ID does not exist" Nov 02 17:53:44.443567 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.443493 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:53:44.443720 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.443675 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aee26f3f-b249-457f-a1e8-77791c926ed8" path="/var/lib/kubelet/pods/aee26f3f-b249-457f-a1e8-77791c926ed8/volumes" Nov 02 17:53:44.444379 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.444358 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b82b62a4-8f2a-498a-99f0-9dea59b90b07" path="/var/lib/kubelet/pods/b82b62a4-8f2a-498a-99f0-9dea59b90b07/volumes" Nov 02 17:53:44.773550 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:53:44.773482 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod341883ea_c8c4_49dd_ba87_ef0a3a539065.slice/crio-81507ef48369f20303ddbd18684e8c9db472e78bbb4c8d9de7f19e773794e31b WatchSource:0}: Error finding container 81507ef48369f20303ddbd18684e8c9db472e78bbb4c8d9de7f19e773794e31b: Status 404 returned error can't find the container with id 81507ef48369f20303ddbd18684e8c9db472e78bbb4c8d9de7f19e773794e31b Nov 02 17:53:44.775617 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.775580 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b"] Nov 02 17:53:44.776969 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.776932 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:53:44.777076 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.777033 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:53:44.777076 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:44.777068 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:53:45.070744 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:45.070657 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" event={"ID":"341883ea-c8c4-49dd-ba87-ef0a3a539065","Type":"ContainerStarted","Data":"b06144af5df7a10b4a043b8b6f8f86b2ab596e60b7a9142453f0d270f511fc67"} Nov 02 17:53:45.070744 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:45.070710 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" event={"ID":"341883ea-c8c4-49dd-ba87-ef0a3a539065","Type":"ContainerStarted","Data":"81507ef48369f20303ddbd18684e8c9db472e78bbb4c8d9de7f19e773794e31b"} Nov 02 17:53:45.119155 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:45.119099 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" podStartSLOduration=2.11908395 podStartE2EDuration="2.11908395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:45.117893875 +0000 UTC m=+1442.813249400" watchObservedRunningTime="2026-11-02 17:53:45.11908395 +0000 UTC m=+1442.814439422" Nov 02 17:53:45.443833 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:45.443751 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:53:45.449296 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:45.449270 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:53:46.078460 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:46.078430 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:53:46.079453 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:46.079430 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:53:47.981133 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:47.981099 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f"] Nov 02 17:53:47.981689 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:47.981389 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" podUID="14703ce4-5dc1-43d0-8368-c217f0bab9eb" containerName="istio-proxy" containerID="cri-o://511355fc6cf5ddf746fdefe2a807368b2ab2c7cc5af36725e04d94cd9ca38119" gracePeriod=30 Nov 02 17:53:48.043785 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:48.043748 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv"] Nov 02 17:53:48.044459 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:48.044403 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" podUID="1ca3f757-188e-4202-9995-fd167353d6be" containerName="istio-proxy" containerID="cri-o://9fa24b625137ca3eabcc24a8a897062c12ffe0d82e560295f2147e7c8fcc9029" gracePeriod=30 Nov 02 17:53:48.843389 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:48.843356 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4"] Nov 02 17:53:48.843631 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:48.843603 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" podUID="006cd169-c635-4f97-a8d1-c21aa6c077c3" containerName="istio-proxy" containerID="cri-o://1f99e421a570c14824d51c908d12e950fa87c327183f1da649ed36b3c285c53d" gracePeriod=30 Nov 02 17:53:49.568001 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:49.567959 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2f6787-77f84bf7c9-bwsjs"] Nov 02 17:53:49.585714 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:49.585681 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6787-77f84bf7c9-bwsjs" Nov 02 17:53:49.595433 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:49.595406 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2f6787-77f84bf7c9-bwsjs"] Nov 02 17:53:49.601710 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:49.601684 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2f6787\"" Nov 02 17:53:49.638885 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:49.638761 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2f6787-dockercfg-w7spt\"" Nov 02 17:53:49.767260 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:49.740702 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl6lr\" (UniqueName: \"kubernetes.io/projected/721435f4-cdae-4071-9767-5dbf79c5dd3f-kube-api-access-bl6lr\") pod \"payload-pre-processing-e2e-inf-a-2f6787-77f84bf7c9-bwsjs\" (UID: \"721435f4-cdae-4071-9767-5dbf79c5dd3f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6787-77f84bf7c9-bwsjs" Nov 02 17:53:49.767821 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:49.767774 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/721435f4-cdae-4071-9767-5dbf79c5dd3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2f6787-77f84bf7c9-bwsjs\" (UID: \"721435f4-cdae-4071-9767-5dbf79c5dd3f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6787-77f84bf7c9-bwsjs" Nov 02 17:53:49.903357 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:49.903248 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl6lr\" (UniqueName: \"kubernetes.io/projected/721435f4-cdae-4071-9767-5dbf79c5dd3f-kube-api-access-bl6lr\") pod \"payload-pre-processing-e2e-inf-a-2f6787-77f84bf7c9-bwsjs\" (UID: \"721435f4-cdae-4071-9767-5dbf79c5dd3f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6787-77f84bf7c9-bwsjs" Nov 02 17:53:49.925445 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:49.925405 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/721435f4-cdae-4071-9767-5dbf79c5dd3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2f6787-77f84bf7c9-bwsjs\" (UID: \"721435f4-cdae-4071-9767-5dbf79c5dd3f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6787-77f84bf7c9-bwsjs" Nov 02 17:53:49.940009 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:49.939977 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/721435f4-cdae-4071-9767-5dbf79c5dd3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2f6787-77f84bf7c9-bwsjs\" (UID: \"721435f4-cdae-4071-9767-5dbf79c5dd3f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6787-77f84bf7c9-bwsjs" Nov 02 17:53:50.041889 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:50.041854 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2f6787-6bb9757b5-tmdf8"] Nov 02 17:53:50.079238 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:50.075883 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2f6787-6bb9757b5-tmdf8"] Nov 02 17:53:50.079238 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:50.075985 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2f6787-6bb9757b5-tmdf8" Nov 02 17:53:50.090331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:50.089660 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl6lr\" (UniqueName: \"kubernetes.io/projected/721435f4-cdae-4071-9767-5dbf79c5dd3f-kube-api-access-bl6lr\") pod \"payload-pre-processing-e2e-inf-a-2f6787-77f84bf7c9-bwsjs\" (UID: \"721435f4-cdae-4071-9767-5dbf79c5dd3f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6787-77f84bf7c9-bwsjs" Nov 02 17:53:50.168156 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:50.168071 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztvbn\" (UniqueName: \"kubernetes.io/projected/8cae7c13-8d21-49f1-b1d4-ea369ec4e54c-kube-api-access-ztvbn\") pod \"payload-processing-e2e-inf-a-2f6787-6bb9757b5-tmdf8\" (UID: \"8cae7c13-8d21-49f1-b1d4-ea369ec4e54c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2f6787-6bb9757b5-tmdf8" Nov 02 17:53:50.168156 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:50.168130 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cae7c13-8d21-49f1-b1d4-ea369ec4e54c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2f6787-6bb9757b5-tmdf8\" (UID: \"8cae7c13-8d21-49f1-b1d4-ea369ec4e54c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2f6787-6bb9757b5-tmdf8" Nov 02 17:53:50.201747 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:50.201717 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6787-77f84bf7c9-bwsjs" Nov 02 17:53:50.270518 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:50.270491 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztvbn\" (UniqueName: \"kubernetes.io/projected/8cae7c13-8d21-49f1-b1d4-ea369ec4e54c-kube-api-access-ztvbn\") pod \"payload-processing-e2e-inf-a-2f6787-6bb9757b5-tmdf8\" (UID: \"8cae7c13-8d21-49f1-b1d4-ea369ec4e54c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2f6787-6bb9757b5-tmdf8" Nov 02 17:53:50.270652 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:50.270542 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cae7c13-8d21-49f1-b1d4-ea369ec4e54c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2f6787-6bb9757b5-tmdf8\" (UID: \"8cae7c13-8d21-49f1-b1d4-ea369ec4e54c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2f6787-6bb9757b5-tmdf8" Nov 02 17:53:50.271253 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:50.271231 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cae7c13-8d21-49f1-b1d4-ea369ec4e54c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2f6787-6bb9757b5-tmdf8\" (UID: \"8cae7c13-8d21-49f1-b1d4-ea369ec4e54c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2f6787-6bb9757b5-tmdf8" Nov 02 17:53:50.295675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:50.295642 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztvbn\" (UniqueName: \"kubernetes.io/projected/8cae7c13-8d21-49f1-b1d4-ea369ec4e54c-kube-api-access-ztvbn\") pod \"payload-processing-e2e-inf-a-2f6787-6bb9757b5-tmdf8\" (UID: \"8cae7c13-8d21-49f1-b1d4-ea369ec4e54c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2f6787-6bb9757b5-tmdf8" Nov 02 17:53:50.396323 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:50.396280 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2f6787-6bb9757b5-tmdf8" Nov 02 17:53:50.523948 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:50.523915 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2f6787-77f84bf7c9-bwsjs"] Nov 02 17:53:50.781903 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:53:50.781858 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cae7c13_8d21_49f1_b1d4_ea369ec4e54c.slice/crio-b65e5a99f5dbbf22aa23c8e044855db5e11a3e37026975436878bbe7c684be21 WatchSource:0}: Error finding container b65e5a99f5dbbf22aa23c8e044855db5e11a3e37026975436878bbe7c684be21: Status 404 returned error can't find the container with id b65e5a99f5dbbf22aa23c8e044855db5e11a3e37026975436878bbe7c684be21 Nov 02 17:53:50.782424 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:50.782278 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2f6787-6bb9757b5-tmdf8"] Nov 02 17:53:51.120277 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:51.120184 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2f6787-6bb9757b5-tmdf8" event={"ID":"8cae7c13-8d21-49f1-b1d4-ea369ec4e54c","Type":"ContainerStarted","Data":"7f61effc758f800483895725f323ac9c517c2158f2ac33abd5bd3cc3fa9a4bea"} Nov 02 17:53:51.120277 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:51.120243 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2f6787-6bb9757b5-tmdf8" event={"ID":"8cae7c13-8d21-49f1-b1d4-ea369ec4e54c","Type":"ContainerStarted","Data":"b65e5a99f5dbbf22aa23c8e044855db5e11a3e37026975436878bbe7c684be21"} Nov 02 17:53:51.120277 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:51.120273 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2f6787-6bb9757b5-tmdf8" Nov 02 17:53:51.122003 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:51.121975 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6787-77f84bf7c9-bwsjs" event={"ID":"721435f4-cdae-4071-9767-5dbf79c5dd3f","Type":"ContainerStarted","Data":"623eafbab03185c9cb82308d3f856ff4f026ffd7af534c76eed53d3f5a130782"} Nov 02 17:53:51.122119 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:51.122022 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6787-77f84bf7c9-bwsjs" event={"ID":"721435f4-cdae-4071-9767-5dbf79c5dd3f","Type":"ContainerStarted","Data":"d9fe3e0a653395bea79f0431d70c509873c67ef2e1fb73985a5d608721d8e038"} Nov 02 17:53:51.122119 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:51.122046 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6787-77f84bf7c9-bwsjs" Nov 02 17:53:51.210643 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:51.210365 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-2f6787-6bb9757b5-tmdf8" podStartSLOduration=2.210344025 podStartE2EDuration="2.210344025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:51.195241483 +0000 UTC m=+1448.890596957" watchObservedRunningTime="2026-11-02 17:53:51.210344025 +0000 UTC m=+1448.905699501" Nov 02 17:53:51.258766 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:51.258737 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q"] Nov 02 17:53:51.258962 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:51.258939 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" podUID="8c8b840f-677a-42c4-b132-7c99567bf1b2" containerName="istio-proxy" containerID="cri-o://0fe944e83c6bcf2338b93ac65a0356312f6ed42614493eb36f55b60a9d140180" gracePeriod=30 Nov 02 17:53:51.259935 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:51.259893 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6787-77f84bf7c9-bwsjs" podStartSLOduration=2.259880341 podStartE2EDuration="2.259880341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:51.257414596 +0000 UTC m=+1448.952770071" watchObservedRunningTime="2026-11-02 17:53:51.259880341 +0000 UTC m=+1448.955235813" Nov 02 17:53:52.605768 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:52.605723 4471 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 17:53:52.606165 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:52.605794 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" podUID="006cd169-c635-4f97-a8d1-c21aa6c077c3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:53:53.156824 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.156785 4471 generic.go:356] "Generic (PLEG): container finished" podID="14703ce4-5dc1-43d0-8368-c217f0bab9eb" containerID="511355fc6cf5ddf746fdefe2a807368b2ab2c7cc5af36725e04d94cd9ca38119" exitCode=0 Nov 02 17:53:53.157029 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.157003 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" event={"ID":"14703ce4-5dc1-43d0-8368-c217f0bab9eb","Type":"ContainerDied","Data":"511355fc6cf5ddf746fdefe2a807368b2ab2c7cc5af36725e04d94cd9ca38119"} Nov 02 17:53:53.157098 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.157073 4471 scope.go:122] "RemoveContainer" containerID="1972f28c65a6080b0770ac769426a03f8ef87c57fca1079e1a53948cfd1550de" Nov 02 17:53:53.361492 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.361469 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:53:53.384502 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.384480 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:53:53.457725 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.457695 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istio-data\") pod \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " Nov 02 17:53:53.457725 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.457734 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istio-envoy\") pod \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " Nov 02 17:53:53.458000 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.457761 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ca3f757-188e-4202-9995-fd167353d6be-istio-token\" (UniqueName: \"kubernetes.io/projected/1ca3f757-188e-4202-9995-fd167353d6be-istio-token\") pod \"1ca3f757-188e-4202-9995-fd167353d6be\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " Nov 02 17:53:53.458000 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.457802 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-workload-socket\") pod \"1ca3f757-188e-4202-9995-fd167353d6be\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " Nov 02 17:53:53.458117 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.458029 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istio-data" pod "14703ce4-5dc1-43d0-8368-c217f0bab9eb" (UID: "14703ce4-5dc1-43d0-8368-c217f0bab9eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:53.458181 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.458117 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-credential-socket\") pod \"1ca3f757-188e-4202-9995-fd167353d6be\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " Nov 02 17:53:53.458181 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.458130 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-workload-socket" pod "1ca3f757-188e-4202-9995-fd167353d6be" (UID: "1ca3f757-188e-4202-9995-fd167353d6be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:53.458288 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.458177 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ca3f757-188e-4202-9995-fd167353d6be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ca3f757-188e-4202-9995-fd167353d6be-istiod-ca-cert\") pod \"1ca3f757-188e-4202-9995-fd167353d6be\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " Nov 02 17:53:53.458288 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.458230 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istio-token\" (UniqueName: \"kubernetes.io/projected/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istio-token\") pod \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " Nov 02 17:53:53.458288 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.458281 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istio-podinfo\") pod \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " Nov 02 17:53:53.458941 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.458330 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-workload-socket\") pod \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " Nov 02 17:53:53.458941 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.458357 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-workload-certs\") pod \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " Nov 02 17:53:53.458941 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.458425 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-istio-data\") pod \"1ca3f757-188e-4202-9995-fd167353d6be\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " Nov 02 17:53:53.458941 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.458433 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-credential-socket" pod "1ca3f757-188e-4202-9995-fd167353d6be" (UID: "1ca3f757-188e-4202-9995-fd167353d6be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:53.458941 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.458466 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ca3f757-188e-4202-9995-fd167353d6be-kube-api-access-qrccr\" (UniqueName: \"kubernetes.io/projected/1ca3f757-188e-4202-9995-fd167353d6be-kube-api-access-qrccr\") pod \"1ca3f757-188e-4202-9995-fd167353d6be\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " Nov 02 17:53:53.458941 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.458568 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-istio-envoy\") pod \"1ca3f757-188e-4202-9995-fd167353d6be\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " Nov 02 17:53:53.458941 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.458650 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-workload-certs\") pod \"1ca3f757-188e-4202-9995-fd167353d6be\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " Nov 02 17:53:53.458941 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.458662 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-workload-socket" pod "14703ce4-5dc1-43d0-8368-c217f0bab9eb" (UID: "14703ce4-5dc1-43d0-8368-c217f0bab9eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:53.458941 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.458703 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istiod-ca-cert\") pod \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " Nov 02 17:53:53.458941 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.458798 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14703ce4-5dc1-43d0-8368-c217f0bab9eb-kube-api-access-77sxd\" (UniqueName: \"kubernetes.io/projected/14703ce4-5dc1-43d0-8368-c217f0bab9eb-kube-api-access-77sxd\") pod \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " Nov 02 17:53:53.458941 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.458861 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-credential-socket\") pod \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\" (UID: \"14703ce4-5dc1-43d0-8368-c217f0bab9eb\") " Nov 02 17:53:53.458941 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.458936 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ca3f757-188e-4202-9995-fd167353d6be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ca3f757-188e-4202-9995-fd167353d6be-istio-podinfo\") pod \"1ca3f757-188e-4202-9995-fd167353d6be\" (UID: \"1ca3f757-188e-4202-9995-fd167353d6be\") " Nov 02 17:53:53.459605 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.459175 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-workload-certs" pod "14703ce4-5dc1-43d0-8368-c217f0bab9eb" (UID: "14703ce4-5dc1-43d0-8368-c217f0bab9eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:53.459605 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.459185 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-workload-certs" pod "1ca3f757-188e-4202-9995-fd167353d6be" (UID: "1ca3f757-188e-4202-9995-fd167353d6be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:53.459605 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.459226 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ca3f757-188e-4202-9995-fd167353d6be-istiod-ca-cert" pod "1ca3f757-188e-4202-9995-fd167353d6be" (UID: "1ca3f757-188e-4202-9995-fd167353d6be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:53.459605 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.459490 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-istio-data" pod "1ca3f757-188e-4202-9995-fd167353d6be" (UID: "1ca3f757-188e-4202-9995-fd167353d6be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:53.460048 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.459886 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.460048 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.459914 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.460048 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.459930 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.460048 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.459943 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.460048 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.460005 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.460048 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.460020 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.460048 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.460020 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-credential-socket" pod "14703ce4-5dc1-43d0-8368-c217f0bab9eb" (UID: "14703ce4-5dc1-43d0-8368-c217f0bab9eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:53.460048 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.460035 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.460048 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.460049 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ca3f757-188e-4202-9995-fd167353d6be-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.460628 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.460359 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istiod-ca-cert" pod "14703ce4-5dc1-43d0-8368-c217f0bab9eb" (UID: "14703ce4-5dc1-43d0-8368-c217f0bab9eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:53.461020 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.460984 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istio-envoy" pod "14703ce4-5dc1-43d0-8368-c217f0bab9eb" (UID: "14703ce4-5dc1-43d0-8368-c217f0bab9eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:53.462198 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.462124 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ca3f757-188e-4202-9995-fd167353d6be-istio-podinfo" pod "1ca3f757-188e-4202-9995-fd167353d6be" (UID: "1ca3f757-188e-4202-9995-fd167353d6be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:53:53.462198 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.462171 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istio-podinfo" pod "14703ce4-5dc1-43d0-8368-c217f0bab9eb" (UID: "14703ce4-5dc1-43d0-8368-c217f0bab9eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:53:53.462638 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.462613 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca3f757-188e-4202-9995-fd167353d6be-istio-token" pod "1ca3f757-188e-4202-9995-fd167353d6be" (UID: "1ca3f757-188e-4202-9995-fd167353d6be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:53.462823 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.462805 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-istio-envoy" pod "1ca3f757-188e-4202-9995-fd167353d6be" (UID: "1ca3f757-188e-4202-9995-fd167353d6be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:53.462875 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.462827 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca3f757-188e-4202-9995-fd167353d6be-kube-api-access-qrccr" pod "1ca3f757-188e-4202-9995-fd167353d6be" (UID: "1ca3f757-188e-4202-9995-fd167353d6be"). InnerVolumeSpecName "kube-api-access-qrccr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:53.463370 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.463354 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istio-token" pod "14703ce4-5dc1-43d0-8368-c217f0bab9eb" (UID: "14703ce4-5dc1-43d0-8368-c217f0bab9eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:53.463773 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.463747 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14703ce4-5dc1-43d0-8368-c217f0bab9eb-kube-api-access-77sxd" pod "14703ce4-5dc1-43d0-8368-c217f0bab9eb" (UID: "14703ce4-5dc1-43d0-8368-c217f0bab9eb"). InnerVolumeSpecName "kube-api-access-77sxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:53.561607 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.561528 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.561607 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.561558 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77sxd\" (UniqueName: \"kubernetes.io/projected/14703ce4-5dc1-43d0-8368-c217f0bab9eb-kube-api-access-77sxd\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.561607 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.561568 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.561607 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.561577 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ca3f757-188e-4202-9995-fd167353d6be-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.561607 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.561585 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.561607 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.561592 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ca3f757-188e-4202-9995-fd167353d6be-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.561607 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.561599 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.561607 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.561607 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14703ce4-5dc1-43d0-8368-c217f0bab9eb-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.562010 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.561615 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrccr\" (UniqueName: \"kubernetes.io/projected/1ca3f757-188e-4202-9995-fd167353d6be-kube-api-access-qrccr\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.562010 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:53.561623 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ca3f757-188e-4202-9995-fd167353d6be-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:54.133124 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.133095 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:53:54.165232 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.165204 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" Nov 02 17:53:54.165437 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.165232 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f" event={"ID":"14703ce4-5dc1-43d0-8368-c217f0bab9eb","Type":"ContainerDied","Data":"c02cc3f9d8a639396d773139a1106693b595895d1155fae053944573f2383493"} Nov 02 17:53:54.165437 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.165295 4471 scope.go:122] "RemoveContainer" containerID="511355fc6cf5ddf746fdefe2a807368b2ab2c7cc5af36725e04d94cd9ca38119" Nov 02 17:53:54.166856 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.166830 4471 generic.go:356] "Generic (PLEG): container finished" podID="1ca3f757-188e-4202-9995-fd167353d6be" containerID="9fa24b625137ca3eabcc24a8a897062c12ffe0d82e560295f2147e7c8fcc9029" exitCode=0 Nov 02 17:53:54.166982 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.166879 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" Nov 02 17:53:54.166982 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.166958 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" event={"ID":"1ca3f757-188e-4202-9995-fd167353d6be","Type":"ContainerDied","Data":"9fa24b625137ca3eabcc24a8a897062c12ffe0d82e560295f2147e7c8fcc9029"} Nov 02 17:53:54.167114 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.167003 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv" event={"ID":"1ca3f757-188e-4202-9995-fd167353d6be","Type":"ContainerDied","Data":"3a43a627b379567403fde8adbd0a70166c0cb3a97e4bbbe4ebc4c8b64bb8488b"} Nov 02 17:53:54.168927 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.168907 4471 generic.go:356] "Generic (PLEG): container finished" podID="006cd169-c635-4f97-a8d1-c21aa6c077c3" containerID="1f99e421a570c14824d51c908d12e950fa87c327183f1da649ed36b3c285c53d" exitCode=0 Nov 02 17:53:54.169055 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.168987 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" Nov 02 17:53:54.169055 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.168991 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" event={"ID":"006cd169-c635-4f97-a8d1-c21aa6c077c3","Type":"ContainerDied","Data":"1f99e421a570c14824d51c908d12e950fa87c327183f1da649ed36b3c285c53d"} Nov 02 17:53:54.169055 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.169041 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4" event={"ID":"006cd169-c635-4f97-a8d1-c21aa6c077c3","Type":"ContainerDied","Data":"0d05c7d1678f808d68e63e396312e0f10bf557f955bc080013e8ba019e8a5247"} Nov 02 17:53:54.171648 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.171624 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/006cd169-c635-4f97-a8d1-c21aa6c077c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006cd169-c635-4f97-a8d1-c21aa6c077c3-istiod-ca-cert\") pod \"006cd169-c635-4f97-a8d1-c21aa6c077c3\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " Nov 02 17:53:54.171764 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.171752 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/006cd169-c635-4f97-a8d1-c21aa6c077c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006cd169-c635-4f97-a8d1-c21aa6c077c3-istio-podinfo\") pod \"006cd169-c635-4f97-a8d1-c21aa6c077c3\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " Nov 02 17:53:54.171822 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.171781 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-istio-envoy\") pod \"006cd169-c635-4f97-a8d1-c21aa6c077c3\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " Nov 02 17:53:54.171878 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.171821 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/006cd169-c635-4f97-a8d1-c21aa6c077c3-kube-api-access-8wc88\" (UniqueName: \"kubernetes.io/projected/006cd169-c635-4f97-a8d1-c21aa6c077c3-kube-api-access-8wc88\") pod \"006cd169-c635-4f97-a8d1-c21aa6c077c3\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " Nov 02 17:53:54.171941 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.171886 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-credential-socket\") pod \"006cd169-c635-4f97-a8d1-c21aa6c077c3\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " Nov 02 17:53:54.172000 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.171967 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-workload-certs\") pod \"006cd169-c635-4f97-a8d1-c21aa6c077c3\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " Nov 02 17:53:54.172056 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.171990 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/006cd169-c635-4f97-a8d1-c21aa6c077c3-istiod-ca-cert" pod "006cd169-c635-4f97-a8d1-c21aa6c077c3" (UID: "006cd169-c635-4f97-a8d1-c21aa6c077c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:54.172192 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.172168 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/006cd169-c635-4f97-a8d1-c21aa6c077c3-istio-token\" (UniqueName: \"kubernetes.io/projected/006cd169-c635-4f97-a8d1-c21aa6c077c3-istio-token\") pod \"006cd169-c635-4f97-a8d1-c21aa6c077c3\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " Nov 02 17:53:54.172330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.172287 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-workload-socket\") pod \"006cd169-c635-4f97-a8d1-c21aa6c077c3\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " Nov 02 17:53:54.172411 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.172341 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-istio-data\") pod \"006cd169-c635-4f97-a8d1-c21aa6c077c3\" (UID: \"006cd169-c635-4f97-a8d1-c21aa6c077c3\") " Nov 02 17:53:54.172411 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.172375 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-credential-socket" pod "006cd169-c635-4f97-a8d1-c21aa6c077c3" (UID: "006cd169-c635-4f97-a8d1-c21aa6c077c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:54.172530 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.172433 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-workload-certs" pod "006cd169-c635-4f97-a8d1-c21aa6c077c3" (UID: "006cd169-c635-4f97-a8d1-c21aa6c077c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:54.172670 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.172648 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-workload-socket" pod "006cd169-c635-4f97-a8d1-c21aa6c077c3" (UID: "006cd169-c635-4f97-a8d1-c21aa6c077c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:54.172931 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.172903 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-istio-data" pod "006cd169-c635-4f97-a8d1-c21aa6c077c3" (UID: "006cd169-c635-4f97-a8d1-c21aa6c077c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:54.173122 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.173097 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:54.173200 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.173128 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:54.173200 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.173146 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006cd169-c635-4f97-a8d1-c21aa6c077c3-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:54.173200 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.173161 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:54.173200 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.173175 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:54.175829 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.174805 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/006cd169-c635-4f97-a8d1-c21aa6c077c3-kube-api-access-8wc88" pod "006cd169-c635-4f97-a8d1-c21aa6c077c3" (UID: "006cd169-c635-4f97-a8d1-c21aa6c077c3"). InnerVolumeSpecName "kube-api-access-8wc88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:54.175829 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.174842 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/006cd169-c635-4f97-a8d1-c21aa6c077c3-istio-token" pod "006cd169-c635-4f97-a8d1-c21aa6c077c3" (UID: "006cd169-c635-4f97-a8d1-c21aa6c077c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:54.175829 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.175786 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-istio-envoy" pod "006cd169-c635-4f97-a8d1-c21aa6c077c3" (UID: "006cd169-c635-4f97-a8d1-c21aa6c077c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:54.176217 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.176192 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/006cd169-c635-4f97-a8d1-c21aa6c077c3-istio-podinfo" pod "006cd169-c635-4f97-a8d1-c21aa6c077c3" (UID: "006cd169-c635-4f97-a8d1-c21aa6c077c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:53:54.179113 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.179098 4471 scope.go:122] "RemoveContainer" containerID="9fa24b625137ca3eabcc24a8a897062c12ffe0d82e560295f2147e7c8fcc9029" Nov 02 17:53:54.216254 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.216224 4471 scope.go:122] "RemoveContainer" containerID="9fa24b625137ca3eabcc24a8a897062c12ffe0d82e560295f2147e7c8fcc9029" Nov 02 17:53:54.216653 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:53:54.216631 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fa24b625137ca3eabcc24a8a897062c12ffe0d82e560295f2147e7c8fcc9029\": container with ID starting with 9fa24b625137ca3eabcc24a8a897062c12ffe0d82e560295f2147e7c8fcc9029 not found: ID does not exist" containerID="9fa24b625137ca3eabcc24a8a897062c12ffe0d82e560295f2147e7c8fcc9029" Nov 02 17:53:54.216748 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.216660 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fa24b625137ca3eabcc24a8a897062c12ffe0d82e560295f2147e7c8fcc9029"} err="failed to get container status \"9fa24b625137ca3eabcc24a8a897062c12ffe0d82e560295f2147e7c8fcc9029\": rpc error: code = NotFound desc = could not find container \"9fa24b625137ca3eabcc24a8a897062c12ffe0d82e560295f2147e7c8fcc9029\": container with ID starting with 9fa24b625137ca3eabcc24a8a897062c12ffe0d82e560295f2147e7c8fcc9029 not found: ID does not exist" Nov 02 17:53:54.216748 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.216680 4471 scope.go:122] "RemoveContainer" containerID="1f99e421a570c14824d51c908d12e950fa87c327183f1da649ed36b3c285c53d" Nov 02 17:53:54.229826 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.229806 4471 scope.go:122] "RemoveContainer" containerID="366bef4b47ff7c4e99b8ebe59103d9643b3e031789c9362d72ef563b40aff5c5" Nov 02 17:53:54.242371 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.242332 4471 scope.go:122] "RemoveContainer" containerID="1f99e421a570c14824d51c908d12e950fa87c327183f1da649ed36b3c285c53d" Nov 02 17:53:54.242673 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:53:54.242652 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f99e421a570c14824d51c908d12e950fa87c327183f1da649ed36b3c285c53d\": container with ID starting with 1f99e421a570c14824d51c908d12e950fa87c327183f1da649ed36b3c285c53d not found: ID does not exist" containerID="1f99e421a570c14824d51c908d12e950fa87c327183f1da649ed36b3c285c53d" Nov 02 17:53:54.242794 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.242680 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f99e421a570c14824d51c908d12e950fa87c327183f1da649ed36b3c285c53d"} err="failed to get container status \"1f99e421a570c14824d51c908d12e950fa87c327183f1da649ed36b3c285c53d\": rpc error: code = NotFound desc = could not find container \"1f99e421a570c14824d51c908d12e950fa87c327183f1da649ed36b3c285c53d\": container with ID starting with 1f99e421a570c14824d51c908d12e950fa87c327183f1da649ed36b3c285c53d not found: ID does not exist" Nov 02 17:53:54.242794 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.242705 4471 scope.go:122] "RemoveContainer" containerID="366bef4b47ff7c4e99b8ebe59103d9643b3e031789c9362d72ef563b40aff5c5" Nov 02 17:53:54.243075 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:53:54.243052 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"366bef4b47ff7c4e99b8ebe59103d9643b3e031789c9362d72ef563b40aff5c5\": container with ID starting with 366bef4b47ff7c4e99b8ebe59103d9643b3e031789c9362d72ef563b40aff5c5 not found: ID does not exist" containerID="366bef4b47ff7c4e99b8ebe59103d9643b3e031789c9362d72ef563b40aff5c5" Nov 02 17:53:54.243138 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.243078 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"366bef4b47ff7c4e99b8ebe59103d9643b3e031789c9362d72ef563b40aff5c5"} err="failed to get container status \"366bef4b47ff7c4e99b8ebe59103d9643b3e031789c9362d72ef563b40aff5c5\": rpc error: code = NotFound desc = could not find container \"366bef4b47ff7c4e99b8ebe59103d9643b3e031789c9362d72ef563b40aff5c5\": container with ID starting with 366bef4b47ff7c4e99b8ebe59103d9643b3e031789c9362d72ef563b40aff5c5 not found: ID does not exist" Nov 02 17:53:54.274959 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.274937 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006cd169-c635-4f97-a8d1-c21aa6c077c3-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:54.274959 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.274960 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006cd169-c635-4f97-a8d1-c21aa6c077c3-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:54.275102 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.274970 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006cd169-c635-4f97-a8d1-c21aa6c077c3-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:54.275102 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.274978 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wc88\" (UniqueName: \"kubernetes.io/projected/006cd169-c635-4f97-a8d1-c21aa6c077c3-kube-api-access-8wc88\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:54.291452 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.291424 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f"] Nov 02 17:53:54.295475 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.295451 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-tk57f"] Nov 02 17:53:54.339631 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.339598 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv"] Nov 02 17:53:54.346536 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.346508 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-84kjv"] Nov 02 17:53:54.442626 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.442556 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14703ce4-5dc1-43d0-8368-c217f0bab9eb" path="/var/lib/kubelet/pods/14703ce4-5dc1-43d0-8368-c217f0bab9eb/volumes" Nov 02 17:53:54.443062 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.443044 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ca3f757-188e-4202-9995-fd167353d6be" path="/var/lib/kubelet/pods/1ca3f757-188e-4202-9995-fd167353d6be/volumes" Nov 02 17:53:54.533324 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.533193 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4"] Nov 02 17:53:54.542719 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:54.542573 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kqdf4"] Nov 02 17:53:56.444162 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:56.444128 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="006cd169-c635-4f97-a8d1-c21aa6c077c3" path="/var/lib/kubelet/pods/006cd169-c635-4f97-a8d1-c21aa6c077c3/volumes" Nov 02 17:53:56.569925 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:56.569865 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:53:56.670012 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:56.669888 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-istio-envoy\") pod \"8c8b840f-677a-42c4-b132-7c99567bf1b2\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " Nov 02 17:53:56.670012 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:56.669975 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-istio-data\") pod \"8c8b840f-677a-42c4-b132-7c99567bf1b2\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " Nov 02 17:53:56.670012 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:56.670012 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-credential-socket\") pod \"8c8b840f-677a-42c4-b132-7c99567bf1b2\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " Nov 02 17:53:56.670289 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:56.670088 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-workload-certs\") pod \"8c8b840f-677a-42c4-b132-7c99567bf1b2\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " Nov 02 17:53:56.670289 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:56.670178 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-workload-socket\") pod \"8c8b840f-677a-42c4-b132-7c99567bf1b2\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " Nov 02 17:53:56.670599 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:56.670569 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-workload-socket" pod "8c8b840f-677a-42c4-b132-7c99567bf1b2" (UID: "8c8b840f-677a-42c4-b132-7c99567bf1b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:56.670735 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:56.670620 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c8b840f-677a-42c4-b132-7c99567bf1b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8b840f-677a-42c4-b132-7c99567bf1b2-istiod-ca-cert\") pod \"8c8b840f-677a-42c4-b132-7c99567bf1b2\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " Nov 02 17:53:56.670735 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:56.670680 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c8b840f-677a-42c4-b132-7c99567bf1b2-kube-api-access-bzkms\" (UniqueName: \"kubernetes.io/projected/8c8b840f-677a-42c4-b132-7c99567bf1b2-kube-api-access-bzkms\") pod \"8c8b840f-677a-42c4-b132-7c99567bf1b2\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " Nov 02 17:53:56.670867 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:56.670847 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c8b840f-677a-42c4-b132-7c99567bf1b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8b840f-677a-42c4-b132-7c99567bf1b2-istio-podinfo\") pod \"8c8b840f-677a-42c4-b132-7c99567bf1b2\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " Nov 02 17:53:56.670929 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:56.670915 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c8b840f-677a-42c4-b132-7c99567bf1b2-istio-token\" (UniqueName: \"kubernetes.io/projected/8c8b840f-677a-42c4-b132-7c99567bf1b2-istio-token\") pod \"8c8b840f-677a-42c4-b132-7c99567bf1b2\" (UID: \"8c8b840f-677a-42c4-b132-7c99567bf1b2\") " Nov 02 17:53:56.671745 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:56.671719 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c8b840f-677a-42c4-b132-7c99567bf1b2-istiod-ca-cert" pod "8c8b840f-677a-42c4-b132-7c99567bf1b2" (UID: "8c8b840f-677a-42c4-b132-7c99567bf1b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:56.671873 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:56.671792 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:56.671999 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:56.671978 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-istio-data" pod "8c8b840f-677a-42c4-b132-7c99567bf1b2" (UID: "8c8b840f-677a-42c4-b132-7c99567bf1b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:56.672054 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:56.672015 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-workload-certs" pod "8c8b840f-677a-42c4-b132-7c99567bf1b2" (UID: "8c8b840f-677a-42c4-b132-7c99567bf1b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:56.672357 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:56.672328 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-credential-socket" pod "8c8b840f-677a-42c4-b132-7c99567bf1b2" (UID: "8c8b840f-677a-42c4-b132-7c99567bf1b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:56.673070 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:56.673043 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-istio-envoy" pod "8c8b840f-677a-42c4-b132-7c99567bf1b2" (UID: "8c8b840f-677a-42c4-b132-7c99567bf1b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:56.673892 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:56.673867 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c8b840f-677a-42c4-b132-7c99567bf1b2-istio-podinfo" pod "8c8b840f-677a-42c4-b132-7c99567bf1b2" (UID: "8c8b840f-677a-42c4-b132-7c99567bf1b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:53:56.674114 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:56.674098 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c8b840f-677a-42c4-b132-7c99567bf1b2-kube-api-access-bzkms" pod "8c8b840f-677a-42c4-b132-7c99567bf1b2" (UID: "8c8b840f-677a-42c4-b132-7c99567bf1b2"). InnerVolumeSpecName "kube-api-access-bzkms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:56.679709 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:56.679683 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c8b840f-677a-42c4-b132-7c99567bf1b2-istio-token" pod "8c8b840f-677a-42c4-b132-7c99567bf1b2" (UID: "8c8b840f-677a-42c4-b132-7c99567bf1b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:56.773500 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:56.773463 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8b840f-677a-42c4-b132-7c99567bf1b2-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:56.773500 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:56.773494 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c8b840f-677a-42c4-b132-7c99567bf1b2-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:56.773500 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:56.773504 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:56.773779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:56.773512 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:56.773779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:56.773520 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:56.773779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:56.773531 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8b840f-677a-42c4-b132-7c99567bf1b2-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:56.773779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:56.773540 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8b840f-677a-42c4-b132-7c99567bf1b2-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:56.773779 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:56.773548 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzkms\" (UniqueName: \"kubernetes.io/projected/8c8b840f-677a-42c4-b132-7c99567bf1b2-kube-api-access-bzkms\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:53:57.211107 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:57.211068 4471 generic.go:356] "Generic (PLEG): container finished" podID="8c8b840f-677a-42c4-b132-7c99567bf1b2" containerID="0fe944e83c6bcf2338b93ac65a0356312f6ed42614493eb36f55b60a9d140180" exitCode=0 Nov 02 17:53:57.211300 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:57.211113 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" Nov 02 17:53:57.211300 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:57.211154 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" event={"ID":"8c8b840f-677a-42c4-b132-7c99567bf1b2","Type":"ContainerDied","Data":"0fe944e83c6bcf2338b93ac65a0356312f6ed42614493eb36f55b60a9d140180"} Nov 02 17:53:57.211300 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:57.211222 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q" event={"ID":"8c8b840f-677a-42c4-b132-7c99567bf1b2","Type":"ContainerDied","Data":"45c44ddf164cb4d6cd5580cd8e0416e5a7e0a19d0a0789f9f6cf93d0b39f16a0"} Nov 02 17:53:57.211300 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:57.211253 4471 scope.go:122] "RemoveContainer" containerID="0fe944e83c6bcf2338b93ac65a0356312f6ed42614493eb36f55b60a9d140180" Nov 02 17:53:57.225317 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:57.225284 4471 scope.go:122] "RemoveContainer" containerID="0fe944e83c6bcf2338b93ac65a0356312f6ed42614493eb36f55b60a9d140180" Nov 02 17:53:57.225596 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:53:57.225573 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fe944e83c6bcf2338b93ac65a0356312f6ed42614493eb36f55b60a9d140180\": container with ID starting with 0fe944e83c6bcf2338b93ac65a0356312f6ed42614493eb36f55b60a9d140180 not found: ID does not exist" containerID="0fe944e83c6bcf2338b93ac65a0356312f6ed42614493eb36f55b60a9d140180" Nov 02 17:53:57.225660 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:57.225608 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fe944e83c6bcf2338b93ac65a0356312f6ed42614493eb36f55b60a9d140180"} err="failed to get container status \"0fe944e83c6bcf2338b93ac65a0356312f6ed42614493eb36f55b60a9d140180\": rpc error: code = NotFound desc = could not find container \"0fe944e83c6bcf2338b93ac65a0356312f6ed42614493eb36f55b60a9d140180\": container with ID starting with 0fe944e83c6bcf2338b93ac65a0356312f6ed42614493eb36f55b60a9d140180 not found: ID does not exist" Nov 02 17:53:57.280519 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:57.280485 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q"] Nov 02 17:53:57.292925 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:57.292895 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-cfj7q"] Nov 02 17:53:58.443146 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:53:58.443111 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c8b840f-677a-42c4-b132-7c99567bf1b2" path="/var/lib/kubelet/pods/8c8b840f-677a-42c4-b132-7c99567bf1b2/volumes" Nov 02 17:54:00.142958 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:00.142923 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6815-b89b00"] Nov 02 17:54:00.143805 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:00.143779 4471 cpu_manager.go:414] "containerMap: removing container" podUID="14703ce4-5dc1-43d0-8368-c217f0bab9eb" containerName="istio-proxy" Nov 02 17:54:00.143907 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:00.143806 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14703ce4-5dc1-43d0-8368-c217f0bab9eb" containerName="istio-proxy" Nov 02 17:54:00.143907 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:00.143831 4471 cpu_manager.go:414] "containerMap: removing container" podUID="14703ce4-5dc1-43d0-8368-c217f0bab9eb" containerName="istio-proxy" Nov 02 17:54:00.143907 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:00.143836 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14703ce4-5dc1-43d0-8368-c217f0bab9eb" containerName="istio-proxy" Nov 02 17:54:00.143907 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:00.143848 4471 cpu_manager.go:414] "containerMap: removing container" podUID="006cd169-c635-4f97-a8d1-c21aa6c077c3" containerName="istio-proxy" Nov 02 17:54:00.143907 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:00.143854 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="006cd169-c635-4f97-a8d1-c21aa6c077c3" containerName="istio-proxy" Nov 02 17:54:00.143907 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:00.143877 4471 cpu_manager.go:414] "containerMap: removing container" podUID="006cd169-c635-4f97-a8d1-c21aa6c077c3" containerName="istio-proxy" Nov 02 17:54:00.143907 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:00.143882 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="006cd169-c635-4f97-a8d1-c21aa6c077c3" containerName="istio-proxy" Nov 02 17:54:00.143907 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:00.143895 4471 cpu_manager.go:414] "containerMap: removing container" podUID="8c8b840f-677a-42c4-b132-7c99567bf1b2" containerName="istio-proxy" Nov 02 17:54:00.143907 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:00.143900 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c8b840f-677a-42c4-b132-7c99567bf1b2" containerName="istio-proxy" Nov 02 17:54:00.144174 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:00.143915 4471 cpu_manager.go:414] "containerMap: removing container" podUID="1ca3f757-188e-4202-9995-fd167353d6be" containerName="istio-proxy" Nov 02 17:54:00.144174 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:00.143920 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ca3f757-188e-4202-9995-fd167353d6be" containerName="istio-proxy" Nov 02 17:54:00.144174 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:00.144022 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="006cd169-c635-4f97-a8d1-c21aa6c077c3" containerName="istio-proxy" Nov 02 17:54:00.144174 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:00.144034 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c8b840f-677a-42c4-b132-7c99567bf1b2" containerName="istio-proxy" Nov 02 17:54:00.144174 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:00.144044 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ca3f757-188e-4202-9995-fd167353d6be" containerName="istio-proxy" Nov 02 17:54:00.144174 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:00.144053 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="14703ce4-5dc1-43d0-8368-c217f0bab9eb" containerName="istio-proxy" Nov 02 17:54:00.149601 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:00.149584 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6815-b89b00" Nov 02 17:54:00.165066 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:00.165043 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9tmjt\"" Nov 02 17:54:00.166357 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:00.166325 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6815-b89b00"] Nov 02 17:54:00.254800 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:00.254769 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nqq5\" (UniqueName: \"kubernetes.io/projected/425958fa-60e0-453e-b4c6-214fa0e3a15c-kube-api-access-9nqq5\") pod \"test-curl-6815-b89b00\" (UID: \"425958fa-60e0-453e-b4c6-214fa0e3a15c\") " pod="opendatahub/test-curl-6815-b89b00" Nov 02 17:54:00.356202 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:00.356169 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nqq5\" (UniqueName: \"kubernetes.io/projected/425958fa-60e0-453e-b4c6-214fa0e3a15c-kube-api-access-9nqq5\") pod \"test-curl-6815-b89b00\" (UID: \"425958fa-60e0-453e-b4c6-214fa0e3a15c\") " pod="opendatahub/test-curl-6815-b89b00" Nov 02 17:54:00.380717 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:00.380690 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nqq5\" (UniqueName: \"kubernetes.io/projected/425958fa-60e0-453e-b4c6-214fa0e3a15c-kube-api-access-9nqq5\") pod \"test-curl-6815-b89b00\" (UID: \"425958fa-60e0-453e-b4c6-214fa0e3a15c\") " pod="opendatahub/test-curl-6815-b89b00" Nov 02 17:54:00.464086 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:00.464012 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6815-b89b00" Nov 02 17:54:00.770431 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:54:00.770394 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod425958fa_60e0_453e_b4c6_214fa0e3a15c.slice/crio-7834cf64f8dae113b14f7d4a6b56ee9b24daee186272b90d5751d98c38108fd9 WatchSource:0}: Error finding container 7834cf64f8dae113b14f7d4a6b56ee9b24daee186272b90d5751d98c38108fd9: Status 404 returned error can't find the container with id 7834cf64f8dae113b14f7d4a6b56ee9b24daee186272b90d5751d98c38108fd9 Nov 02 17:54:00.772820 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:00.772795 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6815-b89b00"] Nov 02 17:54:01.261781 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:01.261750 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6815-b89b00" event={"ID":"425958fa-60e0-453e-b4c6-214fa0e3a15c","Type":"ContainerStarted","Data":"8942dd8fd7a3cc0e613daefb5629999a93b9d4ae841da8e8beb8d5f9ac118d48"} Nov 02 17:54:01.262168 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:01.261799 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6815-b89b00" event={"ID":"425958fa-60e0-453e-b4c6-214fa0e3a15c","Type":"ContainerStarted","Data":"7834cf64f8dae113b14f7d4a6b56ee9b24daee186272b90d5751d98c38108fd9"} Nov 02 17:54:01.301572 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:01.301517 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6815-b89b00" podStartSLOduration=1.301503301 podStartE2EDuration="1.301503301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:01.300634376 +0000 UTC m=+1458.995989852" watchObservedRunningTime="2026-11-02 17:54:01.301503301 +0000 UTC m=+1458.996858773" Nov 02 17:54:02.132723 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:02.132694 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6787-77f84bf7c9-bwsjs" Nov 02 17:54:02.132930 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:02.132763 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2f6787-6bb9757b5-tmdf8" Nov 02 17:54:02.273445 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:02.273413 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6815-b89b00"] Nov 02 17:54:02.279996 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:02.279202 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6815-b89b00"] Nov 02 17:54:02.366110 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:02.366070 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1e5fcb-65b654bb8f-nntvk"] Nov 02 17:54:02.391957 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:02.391884 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1e5fcb-5494fcc9c9-fj45h"] Nov 02 17:54:03.289222 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:03.289153 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6815-b89b00" podUID="425958fa-60e0-453e-b4c6-214fa0e3a15c" containerName="test-curl-6815-b89b00" containerID="cri-o://8942dd8fd7a3cc0e613daefb5629999a93b9d4ae841da8e8beb8d5f9ac118d48" gracePeriod=2 Nov 02 17:54:03.303967 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:54:03.303932 4471 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6815-b89b00\" is forbidden: User \"system:node:ip-10-0-142-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-26.ec2.internal' and this object" podUID="425958fa-60e0-453e-b4c6-214fa0e3a15c" pod="opendatahub/test-curl-6815-b89b00" Nov 02 17:54:05.523623 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:05.523599 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6815-b89b00" Nov 02 17:54:05.538007 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:54:05.537974 4471 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6815-b89b00\" is forbidden: User \"system:node:ip-10-0-142-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-26.ec2.internal' and this object" podUID="425958fa-60e0-453e-b4c6-214fa0e3a15c" pod="opendatahub/test-curl-6815-b89b00" Nov 02 17:54:05.634635 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:05.634561 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/425958fa-60e0-453e-b4c6-214fa0e3a15c-kube-api-access-9nqq5\" (UniqueName: \"kubernetes.io/projected/425958fa-60e0-453e-b4c6-214fa0e3a15c-kube-api-access-9nqq5\") pod \"425958fa-60e0-453e-b4c6-214fa0e3a15c\" (UID: \"425958fa-60e0-453e-b4c6-214fa0e3a15c\") " Nov 02 17:54:05.636642 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:05.636620 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/425958fa-60e0-453e-b4c6-214fa0e3a15c-kube-api-access-9nqq5" pod "425958fa-60e0-453e-b4c6-214fa0e3a15c" (UID: "425958fa-60e0-453e-b4c6-214fa0e3a15c"). InnerVolumeSpecName "kube-api-access-9nqq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:05.736664 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:05.736632 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nqq5\" (UniqueName: \"kubernetes.io/projected/425958fa-60e0-453e-b4c6-214fa0e3a15c-kube-api-access-9nqq5\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:54:06.260108 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.260074 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z"] Nov 02 17:54:06.260971 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.260954 4471 cpu_manager.go:414] "containerMap: removing container" podUID="425958fa-60e0-453e-b4c6-214fa0e3a15c" containerName="test-curl-6815-b89b00" Nov 02 17:54:06.261020 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.260973 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="425958fa-60e0-453e-b4c6-214fa0e3a15c" containerName="test-curl-6815-b89b00" Nov 02 17:54:06.261131 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.261120 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="14703ce4-5dc1-43d0-8368-c217f0bab9eb" containerName="istio-proxy" Nov 02 17:54:06.261175 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.261140 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="425958fa-60e0-453e-b4c6-214fa0e3a15c" containerName="test-curl-6815-b89b00" Nov 02 17:54:06.261175 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.261149 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="006cd169-c635-4f97-a8d1-c21aa6c077c3" containerName="istio-proxy" Nov 02 17:54:06.264027 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.264011 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:54:06.278180 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:54:06.278132 4471 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6815-b89b00\" is forbidden: User \"system:node:ip-10-0-142-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-26.ec2.internal' and this object" podUID="425958fa-60e0-453e-b4c6-214fa0e3a15c" pod="opendatahub/test-curl-6815-b89b00" Nov 02 17:54:06.282979 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.282955 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8"] Nov 02 17:54:06.286499 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.286483 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:54:06.296442 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.296418 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z"] Nov 02 17:54:06.321829 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.315716 4471 generic.go:356] "Generic (PLEG): container finished" podID="425958fa-60e0-453e-b4c6-214fa0e3a15c" containerID="8942dd8fd7a3cc0e613daefb5629999a93b9d4ae841da8e8beb8d5f9ac118d48" exitCode=137 Nov 02 17:54:06.321829 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.315756 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6815-b89b00" Nov 02 17:54:06.321829 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.315769 4471 scope.go:122] "RemoveContainer" containerID="8942dd8fd7a3cc0e613daefb5629999a93b9d4ae841da8e8beb8d5f9ac118d48" Nov 02 17:54:06.330171 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.326375 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8"] Nov 02 17:54:06.337875 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:54:06.337838 4471 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6815-b89b00\" is forbidden: User \"system:node:ip-10-0-142-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-26.ec2.internal' and this object" podUID="425958fa-60e0-453e-b4c6-214fa0e3a15c" pod="opendatahub/test-curl-6815-b89b00" Nov 02 17:54:06.338725 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.338669 4471 scope.go:122] "RemoveContainer" containerID="8942dd8fd7a3cc0e613daefb5629999a93b9d4ae841da8e8beb8d5f9ac118d48" Nov 02 17:54:06.339225 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:54:06.339194 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8942dd8fd7a3cc0e613daefb5629999a93b9d4ae841da8e8beb8d5f9ac118d48\": container with ID starting with 8942dd8fd7a3cc0e613daefb5629999a93b9d4ae841da8e8beb8d5f9ac118d48 not found: ID does not exist" containerID="8942dd8fd7a3cc0e613daefb5629999a93b9d4ae841da8e8beb8d5f9ac118d48" Nov 02 17:54:06.339349 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.339235 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8942dd8fd7a3cc0e613daefb5629999a93b9d4ae841da8e8beb8d5f9ac118d48"} err="failed to get container status \"8942dd8fd7a3cc0e613daefb5629999a93b9d4ae841da8e8beb8d5f9ac118d48\": rpc error: code = NotFound desc = could not find container \"8942dd8fd7a3cc0e613daefb5629999a93b9d4ae841da8e8beb8d5f9ac118d48\": container with ID starting with 8942dd8fd7a3cc0e613daefb5629999a93b9d4ae841da8e8beb8d5f9ac118d48 not found: ID does not exist" Nov 02 17:54:06.351522 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.350738 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/941519c9-0872-4d7c-a5b7-1045b696968a-istio-token\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:54:06.351522 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.350811 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-workload-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:54:06.351522 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.350851 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-workload-certs\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:54:06.351522 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.350910 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-credential-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:54:06.351522 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.350936 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czr8r\" (UniqueName: \"kubernetes.io/projected/941519c9-0872-4d7c-a5b7-1045b696968a-kube-api-access-czr8r\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:54:06.352802 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.352630 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/941519c9-0872-4d7c-a5b7-1045b696968a-istiod-ca-cert\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:54:06.352802 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.352736 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-istio-data\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:54:06.353098 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.352847 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-istio-envoy\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:54:06.353098 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.352888 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/941519c9-0872-4d7c-a5b7-1045b696968a-istio-podinfo\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:54:06.395839 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:54:06.395799 4471 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6815-b89b00\" is forbidden: User \"system:node:ip-10-0-142-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-26.ec2.internal' and this object" podUID="425958fa-60e0-453e-b4c6-214fa0e3a15c" pod="opendatahub/test-curl-6815-b89b00" Nov 02 17:54:06.443940 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.443916 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="425958fa-60e0-453e-b4c6-214fa0e3a15c" path="/var/lib/kubelet/pods/425958fa-60e0-453e-b4c6-214fa0e3a15c/volumes" Nov 02 17:54:06.453857 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.453833 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/941519c9-0872-4d7c-a5b7-1045b696968a-istio-token\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:54:06.453992 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.453879 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-workload-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:54:06.453992 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.453902 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558a5960-22f7-4b6c-a71f-02e532e12576-istio-podinfo\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:54:06.453992 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.453918 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/558a5960-22f7-4b6c-a71f-02e532e12576-istio-token\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:54:06.453992 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.453938 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-workload-certs\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:54:06.453992 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.453954 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-credential-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:54:06.454252 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.454150 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-workload-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:54:06.454252 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.454174 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-credential-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:54:06.454252 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.454228 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czr8r\" (UniqueName: \"kubernetes.io/projected/941519c9-0872-4d7c-a5b7-1045b696968a-kube-api-access-czr8r\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:54:06.454455 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.454335 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-workload-certs\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:54:06.454455 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.454370 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-workload-certs\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:54:06.454455 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.454401 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-workload-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:54:06.454455 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.454412 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-istio-data\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:54:06.454597 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.454511 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/941519c9-0872-4d7c-a5b7-1045b696968a-istiod-ca-cert\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:54:06.454597 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.454537 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558a5960-22f7-4b6c-a71f-02e532e12576-istiod-ca-cert\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:54:06.454597 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.454566 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-credential-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:54:06.454597 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.454574 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-istio-envoy\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:54:06.454826 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.454621 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf4dz\" (UniqueName: \"kubernetes.io/projected/558a5960-22f7-4b6c-a71f-02e532e12576-kube-api-access-vf4dz\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:54:06.454826 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.454686 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-istio-data\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:54:06.454826 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.454809 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-istio-envoy\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:54:06.454971 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.454857 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/941519c9-0872-4d7c-a5b7-1045b696968a-istio-podinfo\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:54:06.454971 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.454923 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-istio-data\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:54:06.455249 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.455226 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/941519c9-0872-4d7c-a5b7-1045b696968a-istiod-ca-cert\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:54:06.457075 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.457043 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-istio-envoy\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:54:06.457269 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.457253 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/941519c9-0872-4d7c-a5b7-1045b696968a-istio-podinfo\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:54:06.478389 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.478365 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/941519c9-0872-4d7c-a5b7-1045b696968a-istio-token\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:54:06.478589 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.478572 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czr8r\" (UniqueName: \"kubernetes.io/projected/941519c9-0872-4d7c-a5b7-1045b696968a-kube-api-access-czr8r\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:54:06.556796 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.556768 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558a5960-22f7-4b6c-a71f-02e532e12576-istiod-ca-cert\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:54:06.557218 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.556836 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-istio-envoy\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:54:06.557218 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.556860 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf4dz\" (UniqueName: \"kubernetes.io/projected/558a5960-22f7-4b6c-a71f-02e532e12576-kube-api-access-vf4dz\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:54:06.557218 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.557091 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558a5960-22f7-4b6c-a71f-02e532e12576-istio-podinfo\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:54:06.557218 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.557136 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/558a5960-22f7-4b6c-a71f-02e532e12576-istio-token\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:54:06.557218 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.557181 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-credential-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:54:06.557539 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.557249 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-workload-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:54:06.557539 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.557324 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-workload-certs\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:54:06.557539 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.557387 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-istio-data\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:54:06.557695 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.557557 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-credential-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:54:06.557695 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.557645 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558a5960-22f7-4b6c-a71f-02e532e12576-istiod-ca-cert\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:54:06.557794 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.557732 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-istio-data\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:54:06.557794 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.557760 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-workload-certs\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:54:06.557873 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.557828 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-workload-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:54:06.559278 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.559258 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-istio-envoy\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:54:06.559520 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.559503 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558a5960-22f7-4b6c-a71f-02e532e12576-istio-podinfo\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:54:06.578830 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.578799 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:54:06.590371 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.590346 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/558a5960-22f7-4b6c-a71f-02e532e12576-istio-token\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:54:06.590498 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.590482 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf4dz\" (UniqueName: \"kubernetes.io/projected/558a5960-22f7-4b6c-a71f-02e532e12576-kube-api-access-vf4dz\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:54:06.602357 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.602332 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:54:06.894181 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:54:06.894154 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod941519c9_0872_4d7c_a5b7_1045b696968a.slice/crio-ef26a1d48ce2840a6f800b27299dbac3006351851c8db9f2c693d30145e96a21 WatchSource:0}: Error finding container ef26a1d48ce2840a6f800b27299dbac3006351851c8db9f2c693d30145e96a21: Status 404 returned error can't find the container with id ef26a1d48ce2840a6f800b27299dbac3006351851c8db9f2c693d30145e96a21 Nov 02 17:54:06.897258 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.897235 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z"] Nov 02 17:54:06.897555 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.897522 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:54:06.897616 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.897592 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:54:06.897653 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.897623 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:54:06.935037 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:54:06.935007 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod558a5960_22f7_4b6c_a71f_02e532e12576.slice/crio-35abe4822ef8c243bab09b85d075c84046c2f973cc6e97149ca94615dc18e526 WatchSource:0}: Error finding container 35abe4822ef8c243bab09b85d075c84046c2f973cc6e97149ca94615dc18e526: Status 404 returned error can't find the container with id 35abe4822ef8c243bab09b85d075c84046c2f973cc6e97149ca94615dc18e526 Nov 02 17:54:06.935571 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.935548 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8"] Nov 02 17:54:06.938348 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.938284 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:54:06.938424 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.938390 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:54:06.938473 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:06.938424 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:54:07.329540 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:07.329502 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" event={"ID":"941519c9-0872-4d7c-a5b7-1045b696968a","Type":"ContainerStarted","Data":"8a1b79bcb54e620302c53dbb71b74f1237e86b76fde13be3fcf19b1343999716"} Nov 02 17:54:07.329835 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:07.329810 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" event={"ID":"941519c9-0872-4d7c-a5b7-1045b696968a","Type":"ContainerStarted","Data":"ef26a1d48ce2840a6f800b27299dbac3006351851c8db9f2c693d30145e96a21"} Nov 02 17:54:07.332257 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:07.332228 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" event={"ID":"558a5960-22f7-4b6c-a71f-02e532e12576","Type":"ContainerStarted","Data":"ed54ab3565c8c5c44023431ced29501851e0290fd81a7dfbc90b10a4676b1450"} Nov 02 17:54:07.332418 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:07.332290 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" event={"ID":"558a5960-22f7-4b6c-a71f-02e532e12576","Type":"ContainerStarted","Data":"35abe4822ef8c243bab09b85d075c84046c2f973cc6e97149ca94615dc18e526"} Nov 02 17:54:07.367386 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:07.367347 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-1e5fcb-65b654bb8f-nntvk" podUID="4af8a256-752b-4927-be4a-71903ce07046" containerName="payload-processing" containerID="cri-o://37b4f4c9cc5cfc242cac5b7289e8902676afb8e0c992841a9bac6b048bd4a426" gracePeriod=25 Nov 02 17:54:07.381120 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:07.381070 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" podStartSLOduration=1.381055234 podStartE2EDuration="1.381055234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:07.37708071 +0000 UTC m=+1465.072436205" watchObservedRunningTime="2026-11-02 17:54:07.381055234 +0000 UTC m=+1465.076410707" Nov 02 17:54:07.392269 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:07.392228 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5fcb-5494fcc9c9-fj45h" podUID="b70c76d6-8ddc-4fc1-85f5-284247309434" containerName="payload-pre-processing" containerID="cri-o://5ab21c97c3dfa9fb2b09960190281044ea2d79b16547b100d9e6c467919f92e8" gracePeriod=25 Nov 02 17:54:07.426075 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:07.426019 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" podStartSLOduration=1.426004045 podStartE2EDuration="1.426004045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:07.4246586 +0000 UTC m=+1465.120014112" watchObservedRunningTime="2026-11-02 17:54:07.426004045 +0000 UTC m=+1465.121359518" Nov 02 17:54:07.619410 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:07.610374 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:54:07.619410 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:07.610442 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:54:07.624210 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:07.624179 4471 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 02 17:54:07.624371 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:07.624236 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" podUID="558a5960-22f7-4b6c-a71f-02e532e12576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 02 17:54:07.624756 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:07.624179 4471 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 17:54:07.624756 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:07.624694 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" podUID="941519c9-0872-4d7c-a5b7-1045b696968a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 17:54:08.137813 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.137786 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1e5fcb-65b654bb8f-nntvk" Nov 02 17:54:08.239968 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.239940 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4af8a256-752b-4927-be4a-71903ce07046-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4af8a256-752b-4927-be4a-71903ce07046-plugins-config-volume\") pod \"4af8a256-752b-4927-be4a-71903ce07046\" (UID: \"4af8a256-752b-4927-be4a-71903ce07046\") " Nov 02 17:54:08.240291 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.240258 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4af8a256-752b-4927-be4a-71903ce07046-plugins-config-volume" pod "4af8a256-752b-4927-be4a-71903ce07046" (UID: "4af8a256-752b-4927-be4a-71903ce07046"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:54:08.240425 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.240401 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4af8a256-752b-4927-be4a-71903ce07046-kube-api-access-ttgc5\" (UniqueName: \"kubernetes.io/projected/4af8a256-752b-4927-be4a-71903ce07046-kube-api-access-ttgc5\") pod \"4af8a256-752b-4927-be4a-71903ce07046\" (UID: \"4af8a256-752b-4927-be4a-71903ce07046\") " Nov 02 17:54:08.240994 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.240971 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4af8a256-752b-4927-be4a-71903ce07046-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:54:08.241796 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.241776 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5fcb-5494fcc9c9-fj45h" Nov 02 17:54:08.242550 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.242530 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4af8a256-752b-4927-be4a-71903ce07046-kube-api-access-ttgc5" pod "4af8a256-752b-4927-be4a-71903ce07046" (UID: "4af8a256-752b-4927-be4a-71903ce07046"). InnerVolumeSpecName "kube-api-access-ttgc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:08.343199 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.343158 4471 generic.go:356] "Generic (PLEG): container finished" podID="4af8a256-752b-4927-be4a-71903ce07046" containerID="37b4f4c9cc5cfc242cac5b7289e8902676afb8e0c992841a9bac6b048bd4a426" exitCode=0 Nov 02 17:54:08.343425 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.343219 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1e5fcb-65b654bb8f-nntvk" Nov 02 17:54:08.343425 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.343265 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b70c76d6-8ddc-4fc1-85f5-284247309434-kube-api-access-68rwf\" (UniqueName: \"kubernetes.io/projected/b70c76d6-8ddc-4fc1-85f5-284247309434-kube-api-access-68rwf\") pod \"b70c76d6-8ddc-4fc1-85f5-284247309434\" (UID: \"b70c76d6-8ddc-4fc1-85f5-284247309434\") " Nov 02 17:54:08.343559 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.343461 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1e5fcb-65b654bb8f-nntvk" event={"ID":"4af8a256-752b-4927-be4a-71903ce07046","Type":"ContainerDied","Data":"37b4f4c9cc5cfc242cac5b7289e8902676afb8e0c992841a9bac6b048bd4a426"} Nov 02 17:54:08.343559 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.343472 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b70c76d6-8ddc-4fc1-85f5-284247309434-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b70c76d6-8ddc-4fc1-85f5-284247309434-plugins-config-volume\") pod \"b70c76d6-8ddc-4fc1-85f5-284247309434\" (UID: \"b70c76d6-8ddc-4fc1-85f5-284247309434\") " Nov 02 17:54:08.343559 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.343514 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1e5fcb-65b654bb8f-nntvk" event={"ID":"4af8a256-752b-4927-be4a-71903ce07046","Type":"ContainerDied","Data":"d82010b3d899254f10159e124fc3c68fd13d4e423d24c3b8b96f12f5373c7fc0"} Nov 02 17:54:08.343559 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.343548 4471 scope.go:122] "RemoveContainer" containerID="37b4f4c9cc5cfc242cac5b7289e8902676afb8e0c992841a9bac6b048bd4a426" Nov 02 17:54:08.344213 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.344186 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttgc5\" (UniqueName: \"kubernetes.io/projected/4af8a256-752b-4927-be4a-71903ce07046-kube-api-access-ttgc5\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:54:08.344619 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.344593 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b70c76d6-8ddc-4fc1-85f5-284247309434-plugins-config-volume" pod "b70c76d6-8ddc-4fc1-85f5-284247309434" (UID: "b70c76d6-8ddc-4fc1-85f5-284247309434"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:54:08.349491 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.349453 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b70c76d6-8ddc-4fc1-85f5-284247309434-kube-api-access-68rwf" pod "b70c76d6-8ddc-4fc1-85f5-284247309434" (UID: "b70c76d6-8ddc-4fc1-85f5-284247309434"). InnerVolumeSpecName "kube-api-access-68rwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:08.352348 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.350638 4471 generic.go:356] "Generic (PLEG): container finished" podID="b70c76d6-8ddc-4fc1-85f5-284247309434" containerID="5ab21c97c3dfa9fb2b09960190281044ea2d79b16547b100d9e6c467919f92e8" exitCode=0 Nov 02 17:54:08.352348 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.351557 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5fcb-5494fcc9c9-fj45h" Nov 02 17:54:08.352570 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.352539 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5fcb-5494fcc9c9-fj45h" event={"ID":"b70c76d6-8ddc-4fc1-85f5-284247309434","Type":"ContainerDied","Data":"5ab21c97c3dfa9fb2b09960190281044ea2d79b16547b100d9e6c467919f92e8"} Nov 02 17:54:08.352636 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.352586 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5fcb-5494fcc9c9-fj45h" event={"ID":"b70c76d6-8ddc-4fc1-85f5-284247309434","Type":"ContainerDied","Data":"6cf9498f660cf07867e67e39f380403a481c4426dce468eb7936f79ffb6630e5"} Nov 02 17:54:08.413790 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.411907 4471 scope.go:122] "RemoveContainer" containerID="37b4f4c9cc5cfc242cac5b7289e8902676afb8e0c992841a9bac6b048bd4a426" Nov 02 17:54:08.425189 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:54:08.425027 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37b4f4c9cc5cfc242cac5b7289e8902676afb8e0c992841a9bac6b048bd4a426\": container with ID starting with 37b4f4c9cc5cfc242cac5b7289e8902676afb8e0c992841a9bac6b048bd4a426 not found: ID does not exist" containerID="37b4f4c9cc5cfc242cac5b7289e8902676afb8e0c992841a9bac6b048bd4a426" Nov 02 17:54:08.425383 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.425209 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37b4f4c9cc5cfc242cac5b7289e8902676afb8e0c992841a9bac6b048bd4a426"} err="failed to get container status \"37b4f4c9cc5cfc242cac5b7289e8902676afb8e0c992841a9bac6b048bd4a426\": rpc error: code = NotFound desc = could not find container \"37b4f4c9cc5cfc242cac5b7289e8902676afb8e0c992841a9bac6b048bd4a426\": container with ID starting with 37b4f4c9cc5cfc242cac5b7289e8902676afb8e0c992841a9bac6b048bd4a426 not found: ID does not exist" Nov 02 17:54:08.425383 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.425238 4471 scope.go:122] "RemoveContainer" containerID="5ab21c97c3dfa9fb2b09960190281044ea2d79b16547b100d9e6c467919f92e8" Nov 02 17:54:08.437530 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:54:08.437452 4471 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af8a256_752b_4927_be4a_71903ce07046.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af8a256_752b_4927_be4a_71903ce07046.slice/crio-d82010b3d899254f10159e124fc3c68fd13d4e423d24c3b8b96f12f5373c7fc0\": RecentStats: unable to find data in memory cache]" Nov 02 17:54:08.548656 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.548630 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68rwf\" (UniqueName: \"kubernetes.io/projected/b70c76d6-8ddc-4fc1-85f5-284247309434-kube-api-access-68rwf\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:54:08.548867 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.548850 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b70c76d6-8ddc-4fc1-85f5-284247309434-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:54:08.587416 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.581755 4471 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 17:54:08.587416 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.581879 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" podUID="941519c9-0872-4d7c-a5b7-1045b696968a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 17:54:08.597534 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.595853 4471 scope.go:122] "RemoveContainer" containerID="5ab21c97c3dfa9fb2b09960190281044ea2d79b16547b100d9e6c467919f92e8" Nov 02 17:54:08.597534 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:54:08.596477 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ab21c97c3dfa9fb2b09960190281044ea2d79b16547b100d9e6c467919f92e8\": container with ID starting with 5ab21c97c3dfa9fb2b09960190281044ea2d79b16547b100d9e6c467919f92e8 not found: ID does not exist" containerID="5ab21c97c3dfa9fb2b09960190281044ea2d79b16547b100d9e6c467919f92e8" Nov 02 17:54:08.597534 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.596515 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ab21c97c3dfa9fb2b09960190281044ea2d79b16547b100d9e6c467919f92e8"} err="failed to get container status \"5ab21c97c3dfa9fb2b09960190281044ea2d79b16547b100d9e6c467919f92e8\": rpc error: code = NotFound desc = could not find container \"5ab21c97c3dfa9fb2b09960190281044ea2d79b16547b100d9e6c467919f92e8\": container with ID starting with 5ab21c97c3dfa9fb2b09960190281044ea2d79b16547b100d9e6c467919f92e8 not found: ID does not exist" Nov 02 17:54:08.612093 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.612052 4471 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 02 17:54:08.612263 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.612106 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" podUID="558a5960-22f7-4b6c-a71f-02e532e12576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 02 17:54:08.638083 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.638054 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1e5fcb-65b654bb8f-nntvk"] Nov 02 17:54:08.644672 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.644647 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1e5fcb-65b654bb8f-nntvk"] Nov 02 17:54:08.711029 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.710984 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1e5fcb-5494fcc9c9-fj45h"] Nov 02 17:54:08.715129 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:08.715103 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1e5fcb-5494fcc9c9-fj45h"] Nov 02 17:54:09.235217 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:54:09.235179 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:54:09.583345 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:09.582854 4471 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 17:54:09.583345 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:09.582913 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" podUID="941519c9-0872-4d7c-a5b7-1045b696968a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 17:54:09.605329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:09.605136 4471 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 02 17:54:09.605329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:09.605193 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" podUID="558a5960-22f7-4b6c-a71f-02e532e12576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 02 17:54:10.443294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:10.443261 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4af8a256-752b-4927-be4a-71903ce07046" path="/var/lib/kubelet/pods/4af8a256-752b-4927-be4a-71903ce07046/volumes" Nov 02 17:54:10.443715 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:10.443701 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b70c76d6-8ddc-4fc1-85f5-284247309434" path="/var/lib/kubelet/pods/b70c76d6-8ddc-4fc1-85f5-284247309434/volumes" Nov 02 17:54:10.588333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:10.587984 4471 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 17:54:10.588333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:10.588063 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" podUID="941519c9-0872-4d7c-a5b7-1045b696968a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 17:54:10.605367 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:10.604770 4471 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 02 17:54:10.605367 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:10.604838 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" podUID="558a5960-22f7-4b6c-a71f-02e532e12576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 02 17:54:11.588335 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:11.587450 4471 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 17:54:11.588335 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:11.587516 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" podUID="941519c9-0872-4d7c-a5b7-1045b696968a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 17:54:11.606136 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:11.606094 4471 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 02 17:54:11.606448 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:11.606404 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" podUID="558a5960-22f7-4b6c-a71f-02e532e12576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 02 17:54:12.581331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:12.580771 4471 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 17:54:12.581331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:12.580838 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" podUID="941519c9-0872-4d7c-a5b7-1045b696968a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 17:54:12.604348 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:12.604281 4471 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 02 17:54:12.604797 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:12.604436 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" podUID="558a5960-22f7-4b6c-a71f-02e532e12576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 02 17:54:13.112854 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:13.112816 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-n5wdd"] Nov 02 17:54:13.638273 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:13.638238 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:54:13.640165 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:13.640141 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:54:13.641905 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:13.641885 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:54:13.664718 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:13.664691 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:54:13.665142 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:13.665117 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:54:13.667946 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:13.667926 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:54:18.113224 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:18.113164 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-n5wdd" podUID="451e2cd1-cdf7-49f8-86a8-dbe0e371b152" containerName="payload-processing" containerID="cri-o://fb82ee62a1068bfa0f5180cac8ee6efcf567c708f5a8993fc8dc00f728fe59b9" gracePeriod=25 Nov 02 17:54:18.392936 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:18.392910 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-n5wdd" Nov 02 17:54:18.479427 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:18.479390 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/451e2cd1-cdf7-49f8-86a8-dbe0e371b152-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/451e2cd1-cdf7-49f8-86a8-dbe0e371b152-plugins-config-volume\") pod \"451e2cd1-cdf7-49f8-86a8-dbe0e371b152\" (UID: \"451e2cd1-cdf7-49f8-86a8-dbe0e371b152\") " Nov 02 17:54:18.479638 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:18.479451 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/451e2cd1-cdf7-49f8-86a8-dbe0e371b152-kube-api-access-wcxhz\" (UniqueName: \"kubernetes.io/projected/451e2cd1-cdf7-49f8-86a8-dbe0e371b152-kube-api-access-wcxhz\") pod \"451e2cd1-cdf7-49f8-86a8-dbe0e371b152\" (UID: \"451e2cd1-cdf7-49f8-86a8-dbe0e371b152\") " Nov 02 17:54:18.479865 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:18.479835 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/451e2cd1-cdf7-49f8-86a8-dbe0e371b152-plugins-config-volume" pod "451e2cd1-cdf7-49f8-86a8-dbe0e371b152" (UID: "451e2cd1-cdf7-49f8-86a8-dbe0e371b152"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:54:18.480414 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:18.480390 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/451e2cd1-cdf7-49f8-86a8-dbe0e371b152-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:54:18.481747 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:18.481722 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/451e2cd1-cdf7-49f8-86a8-dbe0e371b152-kube-api-access-wcxhz" pod "451e2cd1-cdf7-49f8-86a8-dbe0e371b152" (UID: "451e2cd1-cdf7-49f8-86a8-dbe0e371b152"). InnerVolumeSpecName "kube-api-access-wcxhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:18.540276 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:18.540239 4471 generic.go:356] "Generic (PLEG): container finished" podID="451e2cd1-cdf7-49f8-86a8-dbe0e371b152" containerID="fb82ee62a1068bfa0f5180cac8ee6efcf567c708f5a8993fc8dc00f728fe59b9" exitCode=0 Nov 02 17:54:18.540276 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:18.540281 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-n5wdd" event={"ID":"451e2cd1-cdf7-49f8-86a8-dbe0e371b152","Type":"ContainerDied","Data":"fb82ee62a1068bfa0f5180cac8ee6efcf567c708f5a8993fc8dc00f728fe59b9"} Nov 02 17:54:18.540578 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:18.540342 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-n5wdd" Nov 02 17:54:18.540578 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:18.540364 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-n5wdd" event={"ID":"451e2cd1-cdf7-49f8-86a8-dbe0e371b152","Type":"ContainerDied","Data":"9789960bd8210649b33a081f96398d6b7e3e3604d6d76202fdf1784ecda7582b"} Nov 02 17:54:18.540578 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:18.540382 4471 scope.go:122] "RemoveContainer" containerID="fb82ee62a1068bfa0f5180cac8ee6efcf567c708f5a8993fc8dc00f728fe59b9" Nov 02 17:54:18.553745 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:18.553724 4471 scope.go:122] "RemoveContainer" containerID="fb82ee62a1068bfa0f5180cac8ee6efcf567c708f5a8993fc8dc00f728fe59b9" Nov 02 17:54:18.554460 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:54:18.554434 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb82ee62a1068bfa0f5180cac8ee6efcf567c708f5a8993fc8dc00f728fe59b9\": container with ID starting with fb82ee62a1068bfa0f5180cac8ee6efcf567c708f5a8993fc8dc00f728fe59b9 not found: ID does not exist" containerID="fb82ee62a1068bfa0f5180cac8ee6efcf567c708f5a8993fc8dc00f728fe59b9" Nov 02 17:54:18.554626 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:18.554591 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb82ee62a1068bfa0f5180cac8ee6efcf567c708f5a8993fc8dc00f728fe59b9"} err="failed to get container status \"fb82ee62a1068bfa0f5180cac8ee6efcf567c708f5a8993fc8dc00f728fe59b9\": rpc error: code = NotFound desc = could not find container \"fb82ee62a1068bfa0f5180cac8ee6efcf567c708f5a8993fc8dc00f728fe59b9\": container with ID starting with fb82ee62a1068bfa0f5180cac8ee6efcf567c708f5a8993fc8dc00f728fe59b9 not found: ID does not exist" Nov 02 17:54:18.676489 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:18.646658 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcxhz\" (UniqueName: \"kubernetes.io/projected/451e2cd1-cdf7-49f8-86a8-dbe0e371b152-kube-api-access-wcxhz\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:54:18.697412 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:18.697338 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-n5wdd"] Nov 02 17:54:18.717346 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:54:18.717290 4471 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod451e2cd1_cdf7_49f8_86a8_dbe0e371b152.slice/crio-9789960bd8210649b33a081f96398d6b7e3e3604d6d76202fdf1784ecda7582b\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod451e2cd1_cdf7_49f8_86a8_dbe0e371b152.slice\": RecentStats: unable to find data in memory cache]" Nov 02 17:54:18.720996 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:18.719687 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-04e682fe-759db489b7-n5wdd"] Nov 02 17:54:20.445040 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:20.444999 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="451e2cd1-cdf7-49f8-86a8-dbe0e371b152" path="/var/lib/kubelet/pods/451e2cd1-cdf7-49f8-86a8-dbe0e371b152/volumes" Nov 02 17:54:25.516427 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:25.516183 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:25.878169 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:25.877404 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:25.918858 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:25.918440 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:25.962467 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:25.962424 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:26.041126 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:26.041076 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:27.409011 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:27.408971 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl"] Nov 02 17:54:27.409850 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:27.409831 4471 cpu_manager.go:414] "containerMap: removing container" podUID="4af8a256-752b-4927-be4a-71903ce07046" containerName="payload-processing" Nov 02 17:54:27.409850 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:27.409851 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4af8a256-752b-4927-be4a-71903ce07046" containerName="payload-processing" Nov 02 17:54:27.409973 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:27.409892 4471 cpu_manager.go:414] "containerMap: removing container" podUID="b70c76d6-8ddc-4fc1-85f5-284247309434" containerName="payload-pre-processing" Nov 02 17:54:27.409973 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:27.409897 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b70c76d6-8ddc-4fc1-85f5-284247309434" containerName="payload-pre-processing" Nov 02 17:54:27.409973 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:27.409912 4471 cpu_manager.go:414] "containerMap: removing container" podUID="451e2cd1-cdf7-49f8-86a8-dbe0e371b152" containerName="payload-processing" Nov 02 17:54:27.409973 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:27.409918 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="451e2cd1-cdf7-49f8-86a8-dbe0e371b152" containerName="payload-processing" Nov 02 17:54:27.410093 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:27.410040 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="451e2cd1-cdf7-49f8-86a8-dbe0e371b152" containerName="payload-processing" Nov 02 17:54:27.410093 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:27.410054 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="4af8a256-752b-4927-be4a-71903ce07046" containerName="payload-processing" Nov 02 17:54:27.410093 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:27.410067 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="b70c76d6-8ddc-4fc1-85f5-284247309434" containerName="payload-pre-processing" Nov 02 17:54:27.415556 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:27.415531 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl" Nov 02 17:54:27.437959 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:27.437928 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl"] Nov 02 17:54:27.440810 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:27.440785 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-2f6787-serving-cert\"" Nov 02 17:54:27.511939 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:27.511903 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxknd\" (UniqueName: \"kubernetes.io/projected/4cc197cd-1c89-46fc-974b-02d3bf536c3c-kube-api-access-rxknd\") pod \"maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl\" (UID: \"4cc197cd-1c89-46fc-974b-02d3bf536c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl" Nov 02 17:54:27.512106 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:27.511980 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cc197cd-1c89-46fc-974b-02d3bf536c3c-metrics-certs\") pod \"maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl\" (UID: \"4cc197cd-1c89-46fc-974b-02d3bf536c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl" Nov 02 17:54:27.512260 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:27.512238 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cc197cd-1c89-46fc-974b-02d3bf536c3c-maas-api-tls\") pod \"maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl\" (UID: \"4cc197cd-1c89-46fc-974b-02d3bf536c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl" Nov 02 17:54:27.512375 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:27.512346 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cc197cd-1c89-46fc-974b-02d3bf536c3c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl\" (UID: \"4cc197cd-1c89-46fc-974b-02d3bf536c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl" Nov 02 17:54:27.622328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:27.616597 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cc197cd-1c89-46fc-974b-02d3bf536c3c-metrics-certs\") pod \"maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl\" (UID: \"4cc197cd-1c89-46fc-974b-02d3bf536c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl" Nov 02 17:54:27.622328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:27.616829 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cc197cd-1c89-46fc-974b-02d3bf536c3c-maas-api-tls\") pod \"maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl\" (UID: \"4cc197cd-1c89-46fc-974b-02d3bf536c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl" Nov 02 17:54:27.622328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:27.616884 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cc197cd-1c89-46fc-974b-02d3bf536c3c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl\" (UID: \"4cc197cd-1c89-46fc-974b-02d3bf536c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl" Nov 02 17:54:27.622328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:27.617087 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxknd\" (UniqueName: \"kubernetes.io/projected/4cc197cd-1c89-46fc-974b-02d3bf536c3c-kube-api-access-rxknd\") pod \"maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl\" (UID: \"4cc197cd-1c89-46fc-974b-02d3bf536c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl" Nov 02 17:54:27.634768 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:27.626682 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cc197cd-1c89-46fc-974b-02d3bf536c3c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl\" (UID: \"4cc197cd-1c89-46fc-974b-02d3bf536c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl" Nov 02 17:54:27.656521 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:27.656438 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cc197cd-1c89-46fc-974b-02d3bf536c3c-maas-api-tls\") pod \"maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl\" (UID: \"4cc197cd-1c89-46fc-974b-02d3bf536c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl" Nov 02 17:54:27.704971 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:27.704889 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cc197cd-1c89-46fc-974b-02d3bf536c3c-metrics-certs\") pod \"maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl\" (UID: \"4cc197cd-1c89-46fc-974b-02d3bf536c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl" Nov 02 17:54:27.714552 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:27.714360 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxknd\" (UniqueName: \"kubernetes.io/projected/4cc197cd-1c89-46fc-974b-02d3bf536c3c-kube-api-access-rxknd\") pod \"maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl\" (UID: \"4cc197cd-1c89-46fc-974b-02d3bf536c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl" Nov 02 17:54:27.735890 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:27.735605 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl" Nov 02 17:54:28.078884 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:54:28.078855 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc197cd_1c89_46fc_974b_02d3bf536c3c.slice/crio-458fe8f9dec0edea668767cab217a7aee313594119900cb5a14a329433899f2c WatchSource:0}: Error finding container 458fe8f9dec0edea668767cab217a7aee313594119900cb5a14a329433899f2c: Status 404 returned error can't find the container with id 458fe8f9dec0edea668767cab217a7aee313594119900cb5a14a329433899f2c Nov 02 17:54:28.081447 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:28.081422 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl"] Nov 02 17:54:28.846077 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:28.846035 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl" event={"ID":"4cc197cd-1c89-46fc-974b-02d3bf536c3c","Type":"ContainerStarted","Data":"c68848ef4d90b0845b4728d4510abde0a824bd5cc2abbc4293ec04131cc43b61"} Nov 02 17:54:28.846606 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:28.846115 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl" Nov 02 17:54:28.846606 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:28.846140 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl" event={"ID":"4cc197cd-1c89-46fc-974b-02d3bf536c3c","Type":"ContainerStarted","Data":"458fe8f9dec0edea668767cab217a7aee313594119900cb5a14a329433899f2c"} Nov 02 17:54:28.894177 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:28.894122 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl" podStartSLOduration=1.4660596209999999 podStartE2EDuration="1.894106663s" totalImagesPullingTime="428.047042ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:28.890078462 +0000 UTC m=+1486.585433960" watchObservedRunningTime="2026-11-02 17:54:28.894106663 +0000 UTC m=+1486.589462135" Nov 02 17:54:34.859817 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:34.859787 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl" Nov 02 17:54:39.184351 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:54:39.184290 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:54:40.559463 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:40.559396 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8"] Nov 02 17:54:40.704554 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:40.704523 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8"] Nov 02 17:54:40.705070 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:40.704985 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" Nov 02 17:54:40.722826 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:40.722798 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-063f48\"/\"default-dockercfg-bgp7z\"" Nov 02 17:54:40.734862 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:40.734834 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-063f48\"/\"test-model-063f48-kserve-self-signed-certs\"" Nov 02 17:54:40.734984 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:40.734914 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-063f48\"/\"openshift-service-ca.crt\"" Nov 02 17:54:40.735088 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:40.735065 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-063f48\"/\"kube-root-ca.crt\"" Nov 02 17:54:40.852426 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:40.852331 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g862n\" (UniqueName: \"kubernetes.io/projected/198ca016-6343-4dda-999d-63a39d349f3a-kube-api-access-g862n\") pod \"test-model-063f48-kserve-7b74d8c765-jdgd8\" (UID: \"198ca016-6343-4dda-999d-63a39d349f3a\") " pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" Nov 02 17:54:40.852686 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:40.852657 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/198ca016-6343-4dda-999d-63a39d349f3a-tls-certs\") pod \"test-model-063f48-kserve-7b74d8c765-jdgd8\" (UID: \"198ca016-6343-4dda-999d-63a39d349f3a\") " pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" Nov 02 17:54:40.853009 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:40.852966 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/198ca016-6343-4dda-999d-63a39d349f3a-dshm\") pod \"test-model-063f48-kserve-7b74d8c765-jdgd8\" (UID: \"198ca016-6343-4dda-999d-63a39d349f3a\") " pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" Nov 02 17:54:40.853343 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:40.853327 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/198ca016-6343-4dda-999d-63a39d349f3a-tmp-dir\") pod \"test-model-063f48-kserve-7b74d8c765-jdgd8\" (UID: \"198ca016-6343-4dda-999d-63a39d349f3a\") " pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" Nov 02 17:54:40.853488 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:40.853474 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/198ca016-6343-4dda-999d-63a39d349f3a-model-cache\") pod \"test-model-063f48-kserve-7b74d8c765-jdgd8\" (UID: \"198ca016-6343-4dda-999d-63a39d349f3a\") " pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" Nov 02 17:54:40.853670 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:40.853657 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/198ca016-6343-4dda-999d-63a39d349f3a-home\") pod \"test-model-063f48-kserve-7b74d8c765-jdgd8\" (UID: \"198ca016-6343-4dda-999d-63a39d349f3a\") " pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" Nov 02 17:54:40.963330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:40.961599 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/198ca016-6343-4dda-999d-63a39d349f3a-model-cache\") pod \"test-model-063f48-kserve-7b74d8c765-jdgd8\" (UID: \"198ca016-6343-4dda-999d-63a39d349f3a\") " pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" Nov 02 17:54:40.963330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:40.961718 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/198ca016-6343-4dda-999d-63a39d349f3a-home\") pod \"test-model-063f48-kserve-7b74d8c765-jdgd8\" (UID: \"198ca016-6343-4dda-999d-63a39d349f3a\") " pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" Nov 02 17:54:40.963330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:40.961836 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g862n\" (UniqueName: \"kubernetes.io/projected/198ca016-6343-4dda-999d-63a39d349f3a-kube-api-access-g862n\") pod \"test-model-063f48-kserve-7b74d8c765-jdgd8\" (UID: \"198ca016-6343-4dda-999d-63a39d349f3a\") " pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" Nov 02 17:54:40.963330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:40.961871 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/198ca016-6343-4dda-999d-63a39d349f3a-tls-certs\") pod \"test-model-063f48-kserve-7b74d8c765-jdgd8\" (UID: \"198ca016-6343-4dda-999d-63a39d349f3a\") " pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" Nov 02 17:54:40.963330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:40.962026 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/198ca016-6343-4dda-999d-63a39d349f3a-dshm\") pod \"test-model-063f48-kserve-7b74d8c765-jdgd8\" (UID: \"198ca016-6343-4dda-999d-63a39d349f3a\") " pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" Nov 02 17:54:40.963330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:40.962176 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/198ca016-6343-4dda-999d-63a39d349f3a-tmp-dir\") pod \"test-model-063f48-kserve-7b74d8c765-jdgd8\" (UID: \"198ca016-6343-4dda-999d-63a39d349f3a\") " pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" Nov 02 17:54:40.963812 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:40.963349 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/198ca016-6343-4dda-999d-63a39d349f3a-model-cache\") pod \"test-model-063f48-kserve-7b74d8c765-jdgd8\" (UID: \"198ca016-6343-4dda-999d-63a39d349f3a\") " pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" Nov 02 17:54:40.963812 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:40.963593 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/198ca016-6343-4dda-999d-63a39d349f3a-home\") pod \"test-model-063f48-kserve-7b74d8c765-jdgd8\" (UID: \"198ca016-6343-4dda-999d-63a39d349f3a\") " pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" Nov 02 17:54:40.973378 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:40.968524 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/198ca016-6343-4dda-999d-63a39d349f3a-tmp-dir\") pod \"test-model-063f48-kserve-7b74d8c765-jdgd8\" (UID: \"198ca016-6343-4dda-999d-63a39d349f3a\") " pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" Nov 02 17:54:40.973378 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:40.971836 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/198ca016-6343-4dda-999d-63a39d349f3a-dshm\") pod \"test-model-063f48-kserve-7b74d8c765-jdgd8\" (UID: \"198ca016-6343-4dda-999d-63a39d349f3a\") " pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" Nov 02 17:54:41.003281 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:41.003051 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/198ca016-6343-4dda-999d-63a39d349f3a-tls-certs\") pod \"test-model-063f48-kserve-7b74d8c765-jdgd8\" (UID: \"198ca016-6343-4dda-999d-63a39d349f3a\") " pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" Nov 02 17:54:41.010331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:41.004045 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g862n\" (UniqueName: \"kubernetes.io/projected/198ca016-6343-4dda-999d-63a39d349f3a-kube-api-access-g862n\") pod \"test-model-063f48-kserve-7b74d8c765-jdgd8\" (UID: \"198ca016-6343-4dda-999d-63a39d349f3a\") " pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" Nov 02 17:54:41.033460 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:41.033412 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" Nov 02 17:54:41.405547 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:41.405461 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8"] Nov 02 17:54:41.987759 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:41.987722 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" event={"ID":"198ca016-6343-4dda-999d-63a39d349f3a","Type":"ContainerStarted","Data":"dd530edc0c36a79bedce01137e71c6666d838bc7df46d8c76dba442573acb775"} Nov 02 17:54:41.988258 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:41.987774 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" event={"ID":"198ca016-6343-4dda-999d-63a39d349f3a","Type":"ContainerStarted","Data":"029ac69d576c521958922b57048248e7851213caf310ad1e20be9899983b848f"} Nov 02 17:54:42.033176 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:42.033126 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" podStartSLOduration=2.033112235 podStartE2EDuration="2.033112235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:42.030382176 +0000 UTC m=+1499.725737654" watchObservedRunningTime="2026-11-02 17:54:42.033112235 +0000 UTC m=+1499.728467709" Nov 02 17:54:42.034386 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:42.034363 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" Nov 02 17:54:51.034379 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:51.034344 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" Nov 02 17:54:51.049721 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:51.049694 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" Nov 02 17:54:51.098581 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:54:51.098550 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" Nov 02 17:55:02.172386 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:02.151197 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:55:03.345631 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:03.345590 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv"] Nov 02 17:55:03.346028 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:03.345852 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" podUID="7d2e5cde-97d4-4e6a-a564-55693ade99ab" containerName="istio-proxy" containerID="cri-o://f2064eb83d67fba7b2369044bbd4d1e37ee55d5e969026d4d188857c5d6d053b" gracePeriod=30 Nov 02 17:55:06.695260 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:06.695228 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c3c112-6cd749d5cc-mx8nw"] Nov 02 17:55:06.729873 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:06.729842 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c3c112-588f684999-xbtsd"] Nov 02 17:55:08.521221 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:08.521187 4471 generic.go:356] "Generic (PLEG): container finished" podID="7d2e5cde-97d4-4e6a-a564-55693ade99ab" containerID="f2064eb83d67fba7b2369044bbd4d1e37ee55d5e969026d4d188857c5d6d053b" exitCode=0 Nov 02 17:55:08.521649 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:08.521276 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" event={"ID":"7d2e5cde-97d4-4e6a-a564-55693ade99ab","Type":"ContainerDied","Data":"f2064eb83d67fba7b2369044bbd4d1e37ee55d5e969026d4d188857c5d6d053b"} Nov 02 17:55:08.695800 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:08.695770 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:55:08.796475 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:08.796430 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istio-podinfo\") pod \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " Nov 02 17:55:08.796670 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:08.796537 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-workload-socket\") pod \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " Nov 02 17:55:08.796743 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:08.796705 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-credential-socket\") pod \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " Nov 02 17:55:08.796802 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:08.796785 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istio-token\" (UniqueName: \"kubernetes.io/projected/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istio-token\") pod \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " Nov 02 17:55:08.796866 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:08.796817 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d2e5cde-97d4-4e6a-a564-55693ade99ab-kube-api-access-ffr74\" (UniqueName: \"kubernetes.io/projected/7d2e5cde-97d4-4e6a-a564-55693ade99ab-kube-api-access-ffr74\") pod \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " Nov 02 17:55:08.796968 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:08.796947 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istio-data\") pod \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " Nov 02 17:55:08.797033 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:08.796986 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istio-envoy\") pod \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " Nov 02 17:55:08.797095 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:08.797052 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istiod-ca-cert\") pod \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " Nov 02 17:55:08.797172 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:08.797159 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-workload-certs\") pod \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\" (UID: \"7d2e5cde-97d4-4e6a-a564-55693ade99ab\") " Nov 02 17:55:08.798086 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:08.798055 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-workload-certs" pod "7d2e5cde-97d4-4e6a-a564-55693ade99ab" (UID: "7d2e5cde-97d4-4e6a-a564-55693ade99ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:08.798936 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:08.798909 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istiod-ca-cert" pod "7d2e5cde-97d4-4e6a-a564-55693ade99ab" (UID: "7d2e5cde-97d4-4e6a-a564-55693ade99ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:08.799147 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:08.799125 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-credential-socket" pod "7d2e5cde-97d4-4e6a-a564-55693ade99ab" (UID: "7d2e5cde-97d4-4e6a-a564-55693ade99ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:08.799247 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:08.799160 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-workload-socket" pod "7d2e5cde-97d4-4e6a-a564-55693ade99ab" (UID: "7d2e5cde-97d4-4e6a-a564-55693ade99ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:08.799247 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:08.799196 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istio-data" pod "7d2e5cde-97d4-4e6a-a564-55693ade99ab" (UID: "7d2e5cde-97d4-4e6a-a564-55693ade99ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:08.800783 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:08.800757 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istio-podinfo" pod "7d2e5cde-97d4-4e6a-a564-55693ade99ab" (UID: "7d2e5cde-97d4-4e6a-a564-55693ade99ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:55:08.807416 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:08.807391 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d2e5cde-97d4-4e6a-a564-55693ade99ab-kube-api-access-ffr74" pod "7d2e5cde-97d4-4e6a-a564-55693ade99ab" (UID: "7d2e5cde-97d4-4e6a-a564-55693ade99ab"). InnerVolumeSpecName "kube-api-access-ffr74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:08.807765 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:08.807741 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istio-envoy" pod "7d2e5cde-97d4-4e6a-a564-55693ade99ab" (UID: "7d2e5cde-97d4-4e6a-a564-55693ade99ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:08.809485 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:08.809455 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istio-token" pod "7d2e5cde-97d4-4e6a-a564-55693ade99ab" (UID: "7d2e5cde-97d4-4e6a-a564-55693ade99ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:08.899192 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:08.899115 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:08.899192 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:08.899145 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:08.899192 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:08.899155 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffr74\" (UniqueName: \"kubernetes.io/projected/7d2e5cde-97d4-4e6a-a564-55693ade99ab-kube-api-access-ffr74\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:08.899192 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:08.899165 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:08.899192 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:08.899177 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:08.899192 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:08.899188 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:08.899192 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:08.899196 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:08.899555 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:08.899204 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d2e5cde-97d4-4e6a-a564-55693ade99ab-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:08.899555 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:08.899212 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2e5cde-97d4-4e6a-a564-55693ade99ab-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:09.526324 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:55:09.525993 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:55:09.604091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:09.604052 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" Nov 02 17:55:09.604091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:09.604064 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv" event={"ID":"7d2e5cde-97d4-4e6a-a564-55693ade99ab","Type":"ContainerDied","Data":"066a19770511dab549ccb96fc95fe9ad10522dd698f80ec45f44a25b92811f01"} Nov 02 17:55:09.604398 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:09.604137 4471 scope.go:122] "RemoveContainer" containerID="f2064eb83d67fba7b2369044bbd4d1e37ee55d5e969026d4d188857c5d6d053b" Nov 02 17:55:09.694039 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:09.694000 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv"] Nov 02 17:55:09.766324 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:09.766263 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-t5snv"] Nov 02 17:55:10.517059 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:10.489281 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d2e5cde-97d4-4e6a-a564-55693ade99ab" path="/var/lib/kubelet/pods/7d2e5cde-97d4-4e6a-a564-55693ade99ab/volumes" Nov 02 17:55:11.696328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:11.696273 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c3c112-6cd749d5cc-mx8nw" podUID="090b5b1f-22f7-4b8c-acf7-e0c79a2480d1" containerName="payload-processing" containerID="cri-o://a30c47a6ceea5022dda47e137426323f7bad62d3cf7eb8df49e657b1df6c6620" gracePeriod=25 Nov 02 17:55:11.730681 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:11.730642 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3c112-588f684999-xbtsd" podUID="aecb68c6-ddce-402e-9f08-00da13bd48a3" containerName="payload-pre-processing" containerID="cri-o://a9ddb485cc682982e8a8833c79e7ffcb4383cacba2c090ac30efe784561f7d33" gracePeriod=25 Nov 02 17:55:11.922923 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:11.922877 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz"] Nov 02 17:55:11.923126 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:11.923097 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz" podUID="a5271597-990c-4d17-9042-82a6eddada9b" containerName="maas-api" containerID="cri-o://3a30e01022351c1ab7a40f7ad5cf2fce669f8520116b9d75c980c740793ede50" gracePeriod=30 Nov 02 17:55:12.546600 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.546570 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c3c112-6cd749d5cc-mx8nw" Nov 02 17:55:12.548024 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.548000 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3c112-588f684999-xbtsd" Nov 02 17:55:12.620481 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.618960 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aecb68c6-ddce-402e-9f08-00da13bd48a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aecb68c6-ddce-402e-9f08-00da13bd48a3-plugins-config-volume\") pod \"aecb68c6-ddce-402e-9f08-00da13bd48a3\" (UID: \"aecb68c6-ddce-402e-9f08-00da13bd48a3\") " Nov 02 17:55:12.620481 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.619019 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/090b5b1f-22f7-4b8c-acf7-e0c79a2480d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/090b5b1f-22f7-4b8c-acf7-e0c79a2480d1-plugins-config-volume\") pod \"090b5b1f-22f7-4b8c-acf7-e0c79a2480d1\" (UID: \"090b5b1f-22f7-4b8c-acf7-e0c79a2480d1\") " Nov 02 17:55:12.620481 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.619144 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aecb68c6-ddce-402e-9f08-00da13bd48a3-kube-api-access-jsrtp\" (UniqueName: \"kubernetes.io/projected/aecb68c6-ddce-402e-9f08-00da13bd48a3-kube-api-access-jsrtp\") pod \"aecb68c6-ddce-402e-9f08-00da13bd48a3\" (UID: \"aecb68c6-ddce-402e-9f08-00da13bd48a3\") " Nov 02 17:55:12.620481 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.619343 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/090b5b1f-22f7-4b8c-acf7-e0c79a2480d1-kube-api-access-slg67\" (UniqueName: \"kubernetes.io/projected/090b5b1f-22f7-4b8c-acf7-e0c79a2480d1-kube-api-access-slg67\") pod \"090b5b1f-22f7-4b8c-acf7-e0c79a2480d1\" (UID: \"090b5b1f-22f7-4b8c-acf7-e0c79a2480d1\") " Nov 02 17:55:12.695850 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.695747 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/090b5b1f-22f7-4b8c-acf7-e0c79a2480d1-plugins-config-volume" pod "090b5b1f-22f7-4b8c-acf7-e0c79a2480d1" (UID: "090b5b1f-22f7-4b8c-acf7-e0c79a2480d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:12.731800 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.728779 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/090b5b1f-22f7-4b8c-acf7-e0c79a2480d1-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:12.749922 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.749881 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aecb68c6-ddce-402e-9f08-00da13bd48a3-plugins-config-volume" pod "aecb68c6-ddce-402e-9f08-00da13bd48a3" (UID: "aecb68c6-ddce-402e-9f08-00da13bd48a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:12.756778 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.755827 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/090b5b1f-22f7-4b8c-acf7-e0c79a2480d1-kube-api-access-slg67" pod "090b5b1f-22f7-4b8c-acf7-e0c79a2480d1" (UID: "090b5b1f-22f7-4b8c-acf7-e0c79a2480d1"). InnerVolumeSpecName "kube-api-access-slg67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:12.759083 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.759051 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aecb68c6-ddce-402e-9f08-00da13bd48a3-kube-api-access-jsrtp" pod "aecb68c6-ddce-402e-9f08-00da13bd48a3" (UID: "aecb68c6-ddce-402e-9f08-00da13bd48a3"). InnerVolumeSpecName "kube-api-access-jsrtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:12.824875 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.823204 4471 generic.go:356] "Generic (PLEG): container finished" podID="a5271597-990c-4d17-9042-82a6eddada9b" containerID="3a30e01022351c1ab7a40f7ad5cf2fce669f8520116b9d75c980c740793ede50" exitCode=0 Nov 02 17:55:12.824875 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.824816 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz" event={"ID":"a5271597-990c-4d17-9042-82a6eddada9b","Type":"ContainerDied","Data":"3a30e01022351c1ab7a40f7ad5cf2fce669f8520116b9d75c980c740793ede50"} Nov 02 17:55:12.831517 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.831479 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aecb68c6-ddce-402e-9f08-00da13bd48a3-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:12.831517 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.831515 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsrtp\" (UniqueName: \"kubernetes.io/projected/aecb68c6-ddce-402e-9f08-00da13bd48a3-kube-api-access-jsrtp\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:12.831729 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.831532 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slg67\" (UniqueName: \"kubernetes.io/projected/090b5b1f-22f7-4b8c-acf7-e0c79a2480d1-kube-api-access-slg67\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:12.878143 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.853435 4471 generic.go:356] "Generic (PLEG): container finished" podID="aecb68c6-ddce-402e-9f08-00da13bd48a3" containerID="a9ddb485cc682982e8a8833c79e7ffcb4383cacba2c090ac30efe784561f7d33" exitCode=0 Nov 02 17:55:12.878143 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.853609 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3c112-588f684999-xbtsd" event={"ID":"aecb68c6-ddce-402e-9f08-00da13bd48a3","Type":"ContainerDied","Data":"a9ddb485cc682982e8a8833c79e7ffcb4383cacba2c090ac30efe784561f7d33"} Nov 02 17:55:12.878143 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.853673 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3c112-588f684999-xbtsd" event={"ID":"aecb68c6-ddce-402e-9f08-00da13bd48a3","Type":"ContainerDied","Data":"d6f6025da3c5a2c28122212d5cd10148af4ffc0a621c37e31f01f1fa407e5e38"} Nov 02 17:55:12.878143 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.853707 4471 scope.go:122] "RemoveContainer" containerID="a9ddb485cc682982e8a8833c79e7ffcb4383cacba2c090ac30efe784561f7d33" Nov 02 17:55:12.878143 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.853883 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3c112-588f684999-xbtsd" Nov 02 17:55:12.878143 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.874012 4471 generic.go:356] "Generic (PLEG): container finished" podID="090b5b1f-22f7-4b8c-acf7-e0c79a2480d1" containerID="a30c47a6ceea5022dda47e137426323f7bad62d3cf7eb8df49e657b1df6c6620" exitCode=0 Nov 02 17:55:12.878143 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.874112 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c3c112-6cd749d5cc-mx8nw" event={"ID":"090b5b1f-22f7-4b8c-acf7-e0c79a2480d1","Type":"ContainerDied","Data":"a30c47a6ceea5022dda47e137426323f7bad62d3cf7eb8df49e657b1df6c6620"} Nov 02 17:55:12.878143 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.874156 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c3c112-6cd749d5cc-mx8nw" event={"ID":"090b5b1f-22f7-4b8c-acf7-e0c79a2480d1","Type":"ContainerDied","Data":"6117d837810ad290aa4997bbe4c6958f4ed2ce3c6d0b33253f9707a0ad79586e"} Nov 02 17:55:12.878143 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.874239 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c3c112-6cd749d5cc-mx8nw" Nov 02 17:55:12.909585 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.909555 4471 scope.go:122] "RemoveContainer" containerID="a9ddb485cc682982e8a8833c79e7ffcb4383cacba2c090ac30efe784561f7d33" Nov 02 17:55:12.910077 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:55:12.910048 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9ddb485cc682982e8a8833c79e7ffcb4383cacba2c090ac30efe784561f7d33\": container with ID starting with a9ddb485cc682982e8a8833c79e7ffcb4383cacba2c090ac30efe784561f7d33 not found: ID does not exist" containerID="a9ddb485cc682982e8a8833c79e7ffcb4383cacba2c090ac30efe784561f7d33" Nov 02 17:55:12.910160 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.910090 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9ddb485cc682982e8a8833c79e7ffcb4383cacba2c090ac30efe784561f7d33"} err="failed to get container status \"a9ddb485cc682982e8a8833c79e7ffcb4383cacba2c090ac30efe784561f7d33\": rpc error: code = NotFound desc = could not find container \"a9ddb485cc682982e8a8833c79e7ffcb4383cacba2c090ac30efe784561f7d33\": container with ID starting with a9ddb485cc682982e8a8833c79e7ffcb4383cacba2c090ac30efe784561f7d33 not found: ID does not exist" Nov 02 17:55:12.910160 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.910116 4471 scope.go:122] "RemoveContainer" containerID="a30c47a6ceea5022dda47e137426323f7bad62d3cf7eb8df49e657b1df6c6620" Nov 02 17:55:12.921801 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.921718 4471 scope.go:122] "RemoveContainer" containerID="a30c47a6ceea5022dda47e137426323f7bad62d3cf7eb8df49e657b1df6c6620" Nov 02 17:55:12.922161 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:55:12.922102 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a30c47a6ceea5022dda47e137426323f7bad62d3cf7eb8df49e657b1df6c6620\": container with ID starting with a30c47a6ceea5022dda47e137426323f7bad62d3cf7eb8df49e657b1df6c6620 not found: ID does not exist" containerID="a30c47a6ceea5022dda47e137426323f7bad62d3cf7eb8df49e657b1df6c6620" Nov 02 17:55:12.922161 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.922145 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a30c47a6ceea5022dda47e137426323f7bad62d3cf7eb8df49e657b1df6c6620"} err="failed to get container status \"a30c47a6ceea5022dda47e137426323f7bad62d3cf7eb8df49e657b1df6c6620\": rpc error: code = NotFound desc = could not find container \"a30c47a6ceea5022dda47e137426323f7bad62d3cf7eb8df49e657b1df6c6620\": container with ID starting with a30c47a6ceea5022dda47e137426323f7bad62d3cf7eb8df49e657b1df6c6620 not found: ID does not exist" Nov 02 17:55:12.943136 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.942024 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c3c112-588f684999-xbtsd"] Nov 02 17:55:12.947624 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.947564 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c3c112-588f684999-xbtsd"] Nov 02 17:55:12.986108 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.986062 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c3c112-6cd749d5cc-mx8nw"] Nov 02 17:55:12.993824 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.993792 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c3c112-6cd749d5cc-mx8nw"] Nov 02 17:55:12.998821 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:12.998799 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz" Nov 02 17:55:13.034997 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:13.034965 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5271597-990c-4d17-9042-82a6eddada9b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5271597-990c-4d17-9042-82a6eddada9b-maas-api-tls\") pod \"a5271597-990c-4d17-9042-82a6eddada9b\" (UID: \"a5271597-990c-4d17-9042-82a6eddada9b\") " Nov 02 17:55:13.035164 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:13.035012 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5271597-990c-4d17-9042-82a6eddada9b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5271597-990c-4d17-9042-82a6eddada9b-metrics-certs\") pod \"a5271597-990c-4d17-9042-82a6eddada9b\" (UID: \"a5271597-990c-4d17-9042-82a6eddada9b\") " Nov 02 17:55:13.035164 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:13.035072 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5271597-990c-4d17-9042-82a6eddada9b-kube-api-access-jdng2\" (UniqueName: \"kubernetes.io/projected/a5271597-990c-4d17-9042-82a6eddada9b-kube-api-access-jdng2\") pod \"a5271597-990c-4d17-9042-82a6eddada9b\" (UID: \"a5271597-990c-4d17-9042-82a6eddada9b\") " Nov 02 17:55:13.035270 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:13.035243 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5271597-990c-4d17-9042-82a6eddada9b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5271597-990c-4d17-9042-82a6eddada9b-openshift-service-ca\") pod \"a5271597-990c-4d17-9042-82a6eddada9b\" (UID: \"a5271597-990c-4d17-9042-82a6eddada9b\") " Nov 02 17:55:13.035729 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:13.035699 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5271597-990c-4d17-9042-82a6eddada9b-openshift-service-ca" pod "a5271597-990c-4d17-9042-82a6eddada9b" (UID: "a5271597-990c-4d17-9042-82a6eddada9b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:13.036102 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:13.036012 4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5271597-990c-4d17-9042-82a6eddada9b-openshift-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:13.037742 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:13.037714 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5271597-990c-4d17-9042-82a6eddada9b-metrics-certs" pod "a5271597-990c-4d17-9042-82a6eddada9b" (UID: "a5271597-990c-4d17-9042-82a6eddada9b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:55:13.037833 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:13.037755 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5271597-990c-4d17-9042-82a6eddada9b-maas-api-tls" pod "a5271597-990c-4d17-9042-82a6eddada9b" (UID: "a5271597-990c-4d17-9042-82a6eddada9b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:55:13.037961 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:13.037937 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5271597-990c-4d17-9042-82a6eddada9b-kube-api-access-jdng2" pod "a5271597-990c-4d17-9042-82a6eddada9b" (UID: "a5271597-990c-4d17-9042-82a6eddada9b"). InnerVolumeSpecName "kube-api-access-jdng2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:13.138280 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:13.138241 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5271597-990c-4d17-9042-82a6eddada9b-maas-api-tls\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:13.138280 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:13.138277 4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5271597-990c-4d17-9042-82a6eddada9b-metrics-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:13.138509 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:13.138290 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdng2\" (UniqueName: \"kubernetes.io/projected/a5271597-990c-4d17-9042-82a6eddada9b-kube-api-access-jdng2\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:13.886951 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:13.886912 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz" event={"ID":"a5271597-990c-4d17-9042-82a6eddada9b","Type":"ContainerDied","Data":"da1654de8e50dcd97d27da47eda9f293d2cefb59523ac21bc03aed653c0f7e13"} Nov 02 17:55:13.886951 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:13.886939 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz" Nov 02 17:55:13.887494 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:13.886988 4471 scope.go:122] "RemoveContainer" containerID="3a30e01022351c1ab7a40f7ad5cf2fce669f8520116b9d75c980c740793ede50" Nov 02 17:55:13.944884 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:13.944851 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz"] Nov 02 17:55:13.963105 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:13.963067 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz"] Nov 02 17:55:14.444498 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:14.444461 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="090b5b1f-22f7-4b8c-acf7-e0c79a2480d1" path="/var/lib/kubelet/pods/090b5b1f-22f7-4b8c-acf7-e0c79a2480d1/volumes" Nov 02 17:55:14.445251 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:14.445226 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5271597-990c-4d17-9042-82a6eddada9b" path="/var/lib/kubelet/pods/a5271597-990c-4d17-9042-82a6eddada9b/volumes" Nov 02 17:55:14.445878 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:14.445848 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aecb68c6-ddce-402e-9f08-00da13bd48a3" path="/var/lib/kubelet/pods/aecb68c6-ddce-402e-9f08-00da13bd48a3/volumes" Nov 02 17:55:15.521520 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.521484 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s"] Nov 02 17:55:15.522721 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.522699 4471 cpu_manager.go:414] "containerMap: removing container" podUID="7d2e5cde-97d4-4e6a-a564-55693ade99ab" containerName="istio-proxy" Nov 02 17:55:15.522825 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.522722 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d2e5cde-97d4-4e6a-a564-55693ade99ab" containerName="istio-proxy" Nov 02 17:55:15.522825 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.522756 4471 cpu_manager.go:414] "containerMap: removing container" podUID="a5271597-990c-4d17-9042-82a6eddada9b" containerName="maas-api" Nov 02 17:55:15.522825 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.522764 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5271597-990c-4d17-9042-82a6eddada9b" containerName="maas-api" Nov 02 17:55:15.522825 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.522780 4471 cpu_manager.go:414] "containerMap: removing container" podUID="aecb68c6-ddce-402e-9f08-00da13bd48a3" containerName="payload-pre-processing" Nov 02 17:55:15.522825 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.522787 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aecb68c6-ddce-402e-9f08-00da13bd48a3" containerName="payload-pre-processing" Nov 02 17:55:15.523077 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.522879 4471 cpu_manager.go:414] "containerMap: removing container" podUID="090b5b1f-22f7-4b8c-acf7-e0c79a2480d1" containerName="payload-processing" Nov 02 17:55:15.523077 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.522887 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="090b5b1f-22f7-4b8c-acf7-e0c79a2480d1" containerName="payload-processing" Nov 02 17:55:15.523184 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.523085 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="090b5b1f-22f7-4b8c-acf7-e0c79a2480d1" containerName="payload-processing" Nov 02 17:55:15.523184 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.523102 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d2e5cde-97d4-4e6a-a564-55693ade99ab" containerName="istio-proxy" Nov 02 17:55:15.523184 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.523118 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5271597-990c-4d17-9042-82a6eddada9b" containerName="maas-api" Nov 02 17:55:15.523184 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.523128 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="aecb68c6-ddce-402e-9f08-00da13bd48a3" containerName="payload-pre-processing" Nov 02 17:55:15.526714 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.526691 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:55:15.553289 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.553260 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s"] Nov 02 17:55:15.560608 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.560581 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8a209f-openshift-default-dockercfg-whvdh\"" Nov 02 17:55:15.603508 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.603478 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a819a2f-01ae-45ba-8c60-f3382617c58c-istio-podinfo\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:55:15.603665 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.603626 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-credential-socket\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:55:15.603665 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.603660 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-workload-socket\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:55:15.603790 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.603756 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2zdz\" (UniqueName: \"kubernetes.io/projected/5a819a2f-01ae-45ba-8c60-f3382617c58c-kube-api-access-t2zdz\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:55:15.603924 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.603908 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a819a2f-01ae-45ba-8c60-f3382617c58c-istiod-ca-cert\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:55:15.603961 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.603943 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-istio-data\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:55:15.603995 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.603964 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-istio-envoy\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:55:15.604057 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.604042 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a819a2f-01ae-45ba-8c60-f3382617c58c-istio-token\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:55:15.604178 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.604160 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-workload-certs\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:55:15.720337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.714519 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a819a2f-01ae-45ba-8c60-f3382617c58c-istiod-ca-cert\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:55:15.720337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.714589 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-istio-data\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:55:15.720337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.714625 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-istio-envoy\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:55:15.720337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.714666 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a819a2f-01ae-45ba-8c60-f3382617c58c-istio-token\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:55:15.720337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.714775 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-workload-certs\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:55:15.720337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.714841 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a819a2f-01ae-45ba-8c60-f3382617c58c-istio-podinfo\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:55:15.720337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.715175 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-credential-socket\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:55:15.720337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.715215 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-workload-socket\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:55:15.720337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.715259 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2zdz\" (UniqueName: \"kubernetes.io/projected/5a819a2f-01ae-45ba-8c60-f3382617c58c-kube-api-access-t2zdz\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:55:15.720337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.716687 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-workload-certs\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:55:15.720337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.716965 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-istio-data\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:55:15.720337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.717805 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-credential-socket\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:55:15.720337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.718473 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-workload-socket\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:55:15.745331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.736707 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-istio-envoy\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:55:15.745331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.741603 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a819a2f-01ae-45ba-8c60-f3382617c58c-istiod-ca-cert\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:55:15.756366 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.753386 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2zdz\" (UniqueName: \"kubernetes.io/projected/5a819a2f-01ae-45ba-8c60-f3382617c58c-kube-api-access-t2zdz\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:55:15.762327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.760470 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5fcb-854944c5db-2mqwz" podUID="a5271597-990c-4d17-9042-82a6eddada9b" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.128:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:15.771358 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.768921 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a819a2f-01ae-45ba-8c60-f3382617c58c-istio-podinfo\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:55:15.777368 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.776882 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a819a2f-01ae-45ba-8c60-f3382617c58c-istio-token\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:55:15.853325 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.852891 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:55:15.997786 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.997298 4471 generic.go:356] "Generic (PLEG): container finished" podID="341883ea-c8c4-49dd-ba87-ef0a3a539065" containerID="b06144af5df7a10b4a043b8b6f8f86b2ab596e60b7a9142453f0d270f511fc67" exitCode=0 Nov 02 17:55:15.997786 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.997622 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" event={"ID":"341883ea-c8c4-49dd-ba87-ef0a3a539065","Type":"ContainerDied","Data":"b06144af5df7a10b4a043b8b6f8f86b2ab596e60b7a9142453f0d270f511fc67"} Nov 02 17:55:15.998183 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:15.998161 4471 scope.go:122] "RemoveContainer" containerID="b06144af5df7a10b4a043b8b6f8f86b2ab596e60b7a9142453f0d270f511fc67" Nov 02 17:55:16.000957 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:16.000921 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:55:16.001183 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:16.001159 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:55:16.001411 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:16.001390 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:55:16.090488 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:16.090411 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:55:16.364656 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:55:16.364625 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a819a2f_01ae_45ba_8c60_f3382617c58c.slice/crio-cbb8a64632149e2f4105fb5f3e49d1d7d0089d5f951baf0ba26536aba2adf86c WatchSource:0}: Error finding container cbb8a64632149e2f4105fb5f3e49d1d7d0089d5f951baf0ba26536aba2adf86c: Status 404 returned error can't find the container with id cbb8a64632149e2f4105fb5f3e49d1d7d0089d5f951baf0ba26536aba2adf86c Nov 02 17:55:16.365791 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:16.365755 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s"] Nov 02 17:55:16.368936 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:16.368891 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:55:16.369057 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:16.368989 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:55:16.369057 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:16.369036 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:55:17.007463 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:17.007424 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" event={"ID":"341883ea-c8c4-49dd-ba87-ef0a3a539065","Type":"ContainerStarted","Data":"223b0b7f02faec9930397bf9dd171da4d453cca519329ca75cff4e1ce02d36de"} Nov 02 17:55:17.009447 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:17.009422 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" event={"ID":"5a819a2f-01ae-45ba-8c60-f3382617c58c","Type":"ContainerStarted","Data":"ded9c25461a48a974f0c690ed9509607e191a2124ad1cd4bec1c3bea33eab9aa"} Nov 02 17:55:17.009607 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:17.009468 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" event={"ID":"5a819a2f-01ae-45ba-8c60-f3382617c58c","Type":"ContainerStarted","Data":"cbb8a64632149e2f4105fb5f3e49d1d7d0089d5f951baf0ba26536aba2adf86c"} Nov 02 17:55:17.443697 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:17.443663 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:55:17.445097 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:17.445071 4471 patch_prober.go:29] interesting pod/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 02 17:55:17.445224 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:17.445115 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" podUID="341883ea-c8c4-49dd-ba87-ef0a3a539065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 02 17:55:17.853863 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:17.853828 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:55:17.877329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:17.876602 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:55:17.951678 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:17.951625 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" podStartSLOduration=2.951608995 podStartE2EDuration="2.951608995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:17.179688669 +0000 UTC m=+1534.875044148" watchObservedRunningTime="2026-11-02 17:55:17.951608995 +0000 UTC m=+1535.646964524" Nov 02 17:55:18.016947 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:18.016906 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:55:18.017793 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:18.017774 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:55:18.443985 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:18.443953 4471 patch_prober.go:29] interesting pod/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 02 17:55:18.444164 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:18.444005 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" podUID="341883ea-c8c4-49dd-ba87-ef0a3a539065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 02 17:55:19.444156 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:19.444127 4471 patch_prober.go:29] interesting pod/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 02 17:55:19.444559 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:19.444186 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" podUID="341883ea-c8c4-49dd-ba87-ef0a3a539065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 02 17:55:20.187711 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:20.187675 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55"] Nov 02 17:55:20.187933 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:20.187903 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" podUID="d505d54d-e785-4f07-87c1-7aef49f974a3" containerName="istio-proxy" containerID="cri-o://be9bb81b215f788abfab429f970bb3c8b80255b51a78eab8169717feee0c5673" gracePeriod=30 Nov 02 17:55:20.453397 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:20.453296 4471 patch_prober.go:29] interesting pod/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 02 17:55:20.455629 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:20.455597 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" podUID="341883ea-c8c4-49dd-ba87-ef0a3a539065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 02 17:55:21.444231 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:21.444196 4471 patch_prober.go:29] interesting pod/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 02 17:55:21.444481 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:21.444255 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" podUID="341883ea-c8c4-49dd-ba87-ef0a3a539065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 02 17:55:21.991930 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:21.991890 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8a209f-789b5bc59f-lgg4z"] Nov 02 17:55:21.997981 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:21.997953 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a209f-789b5bc59f-lgg4z" Nov 02 17:55:22.015357 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:22.015327 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8a209f-789b5bc59f-lgg4z"] Nov 02 17:55:22.047297 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:22.047262 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8a209f\"" Nov 02 17:55:22.047520 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:22.047435 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8a209f-dockercfg-g8ztf\"" Nov 02 17:55:22.214331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:22.209718 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2eb7cb-d5ff-4351-84ac-f19fabf7e616-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8a209f-789b5bc59f-lgg4z\" (UID: \"dd2eb7cb-d5ff-4351-84ac-f19fabf7e616\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a209f-789b5bc59f-lgg4z" Nov 02 17:55:22.214331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:22.211095 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kvc7\" (UniqueName: \"kubernetes.io/projected/dd2eb7cb-d5ff-4351-84ac-f19fabf7e616-kube-api-access-8kvc7\") pod \"payload-pre-processing-e2e-inf-b-8a209f-789b5bc59f-lgg4z\" (UID: \"dd2eb7cb-d5ff-4351-84ac-f19fabf7e616\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a209f-789b5bc59f-lgg4z" Nov 02 17:55:22.334329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:22.324076 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2eb7cb-d5ff-4351-84ac-f19fabf7e616-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8a209f-789b5bc59f-lgg4z\" (UID: \"dd2eb7cb-d5ff-4351-84ac-f19fabf7e616\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a209f-789b5bc59f-lgg4z" Nov 02 17:55:22.334329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:22.324341 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kvc7\" (UniqueName: \"kubernetes.io/projected/dd2eb7cb-d5ff-4351-84ac-f19fabf7e616-kube-api-access-8kvc7\") pod \"payload-pre-processing-e2e-inf-b-8a209f-789b5bc59f-lgg4z\" (UID: \"dd2eb7cb-d5ff-4351-84ac-f19fabf7e616\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a209f-789b5bc59f-lgg4z" Nov 02 17:55:22.362330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:22.361848 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2eb7cb-d5ff-4351-84ac-f19fabf7e616-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8a209f-789b5bc59f-lgg4z\" (UID: \"dd2eb7cb-d5ff-4351-84ac-f19fabf7e616\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a209f-789b5bc59f-lgg4z" Nov 02 17:55:22.470722 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:22.470683 4471 patch_prober.go:29] interesting pod/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 02 17:55:22.611428 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:22.572457 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" podUID="341883ea-c8c4-49dd-ba87-ef0a3a539065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 02 17:55:22.611428 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:22.568113 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kvc7\" (UniqueName: \"kubernetes.io/projected/dd2eb7cb-d5ff-4351-84ac-f19fabf7e616-kube-api-access-8kvc7\") pod \"payload-pre-processing-e2e-inf-b-8a209f-789b5bc59f-lgg4z\" (UID: \"dd2eb7cb-d5ff-4351-84ac-f19fabf7e616\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a209f-789b5bc59f-lgg4z" Nov 02 17:55:22.633006 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:22.616482 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a209f-789b5bc59f-lgg4z" Nov 02 17:55:23.005843 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:55:23.005812 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd2eb7cb_d5ff_4351_84ac_f19fabf7e616.slice/crio-05c48c7ddda25695f422d937e1bb97394dc591cc69333d45cc7c1ad8d43ec25f WatchSource:0}: Error finding container 05c48c7ddda25695f422d937e1bb97394dc591cc69333d45cc7c1ad8d43ec25f: Status 404 returned error can't find the container with id 05c48c7ddda25695f422d937e1bb97394dc591cc69333d45cc7c1ad8d43ec25f Nov 02 17:55:23.008291 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:23.008265 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8a209f-789b5bc59f-lgg4z"] Nov 02 17:55:23.346576 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:23.346531 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a209f-789b5bc59f-lgg4z" event={"ID":"dd2eb7cb-d5ff-4351-84ac-f19fabf7e616","Type":"ContainerStarted","Data":"47345402a5b9cb8bb2b10d5ba64e407fc614e603dc190b7b317dbefb51ca51a6"} Nov 02 17:55:23.346739 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:23.346595 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a209f-789b5bc59f-lgg4z" event={"ID":"dd2eb7cb-d5ff-4351-84ac-f19fabf7e616","Type":"ContainerStarted","Data":"05c48c7ddda25695f422d937e1bb97394dc591cc69333d45cc7c1ad8d43ec25f"} Nov 02 17:55:23.346739 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:23.346627 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a209f-789b5bc59f-lgg4z" Nov 02 17:55:23.392130 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:23.392078 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a209f-789b5bc59f-lgg4z" podStartSLOduration=2.392063655 podStartE2EDuration="2.392063655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:23.388870006 +0000 UTC m=+1541.084225503" watchObservedRunningTime="2026-11-02 17:55:23.392063655 +0000 UTC m=+1541.087419171" Nov 02 17:55:23.448875 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:23.448844 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:55:23.449419 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:23.449396 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:55:23.450215 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:23.450197 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 17:55:25.255490 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.255454 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs"] Nov 02 17:55:25.263976 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.263949 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:55:25.287479 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.286827 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs"] Nov 02 17:55:25.297149 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.297109 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwhbk\" (UniqueName: \"kubernetes.io/projected/19632b35-0e41-4d90-89c3-ced789733fbc-kube-api-access-vwhbk\") pod \"e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:55:25.297291 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.297202 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19632b35-0e41-4d90-89c3-ced789733fbc-istio-podinfo\") pod \"e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:55:25.297380 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.297347 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-credential-socket\") pod \"e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:55:25.297494 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.297380 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19632b35-0e41-4d90-89c3-ced789733fbc-istiod-ca-cert\") pod \"e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:55:25.297494 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.297441 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19632b35-0e41-4d90-89c3-ced789733fbc-istio-token\") pod \"e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:55:25.297494 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.297467 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-workload-certs\") pod \"e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:55:25.297494 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.297491 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-workload-socket\") pod \"e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:55:25.297701 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.297538 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-istio-envoy\") pod \"e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:55:25.297701 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.297654 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-istio-data\") pod \"e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:55:25.298031 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.298011 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-bb6d2e-openshift-default-dockercfg-td5t7\"" Nov 02 17:55:25.442330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.429793 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-credential-socket\") pod \"e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:55:25.442330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.429848 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19632b35-0e41-4d90-89c3-ced789733fbc-istiod-ca-cert\") pod \"e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:55:25.442330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.429947 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19632b35-0e41-4d90-89c3-ced789733fbc-istio-token\") pod \"e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:55:25.442330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.429977 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-workload-certs\") pod \"e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:55:25.442330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.430008 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-workload-socket\") pod \"e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:55:25.442330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.430063 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-istio-envoy\") pod \"e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:55:25.442330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.430210 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-istio-data\") pod \"e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:55:25.442330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.430281 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwhbk\" (UniqueName: \"kubernetes.io/projected/19632b35-0e41-4d90-89c3-ced789733fbc-kube-api-access-vwhbk\") pod \"e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:55:25.442330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.430393 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19632b35-0e41-4d90-89c3-ced789733fbc-istio-podinfo\") pod \"e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:55:25.442330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.440010 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-workload-certs\") pod \"e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:55:25.484393 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.453561 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-istio-data\") pod \"e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:55:25.484393 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.454088 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-workload-socket\") pod \"e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:55:25.503332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.490646 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-istio-envoy\") pod \"e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:55:25.591984 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.590492 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-credential-socket\") pod \"e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:55:25.712337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.700249 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19632b35-0e41-4d90-89c3-ced789733fbc-istio-podinfo\") pod \"e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:55:25.712337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.703520 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19632b35-0e41-4d90-89c3-ced789733fbc-istiod-ca-cert\") pod \"e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:55:25.712718 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.712685 4471 generic.go:356] "Generic (PLEG): container finished" podID="d505d54d-e785-4f07-87c1-7aef49f974a3" containerID="be9bb81b215f788abfab429f970bb3c8b80255b51a78eab8169717feee0c5673" exitCode=0 Nov 02 17:55:25.713962 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.713931 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" event={"ID":"d505d54d-e785-4f07-87c1-7aef49f974a3","Type":"ContainerDied","Data":"be9bb81b215f788abfab429f970bb3c8b80255b51a78eab8169717feee0c5673"} Nov 02 17:55:25.741075 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.740998 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19632b35-0e41-4d90-89c3-ced789733fbc-istio-token\") pod \"e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:55:25.748957 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.748891 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwhbk\" (UniqueName: \"kubernetes.io/projected/19632b35-0e41-4d90-89c3-ced789733fbc-kube-api-access-vwhbk\") pod \"e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:55:25.827354 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.827288 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:55:25.958192 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.957075 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:55:25.960667 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.958675 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-istio-envoy\") pod \"d505d54d-e785-4f07-87c1-7aef49f974a3\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " Nov 02 17:55:25.960667 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.958738 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d505d54d-e785-4f07-87c1-7aef49f974a3-kube-api-access-n2b4h\" (UniqueName: \"kubernetes.io/projected/d505d54d-e785-4f07-87c1-7aef49f974a3-kube-api-access-n2b4h\") pod \"d505d54d-e785-4f07-87c1-7aef49f974a3\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " Nov 02 17:55:25.960667 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.958863 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d505d54d-e785-4f07-87c1-7aef49f974a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d505d54d-e785-4f07-87c1-7aef49f974a3-istio-podinfo\") pod \"d505d54d-e785-4f07-87c1-7aef49f974a3\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " Nov 02 17:55:25.960667 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.958890 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d505d54d-e785-4f07-87c1-7aef49f974a3-istio-token\" (UniqueName: \"kubernetes.io/projected/d505d54d-e785-4f07-87c1-7aef49f974a3-istio-token\") pod \"d505d54d-e785-4f07-87c1-7aef49f974a3\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " Nov 02 17:55:25.960667 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.959089 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-workload-certs\") pod \"d505d54d-e785-4f07-87c1-7aef49f974a3\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " Nov 02 17:55:25.960667 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.959135 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-credential-socket\") pod \"d505d54d-e785-4f07-87c1-7aef49f974a3\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " Nov 02 17:55:25.960667 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.960488 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-workload-certs" pod "d505d54d-e785-4f07-87c1-7aef49f974a3" (UID: "d505d54d-e785-4f07-87c1-7aef49f974a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:25.967151 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.967114 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-credential-socket" pod "d505d54d-e785-4f07-87c1-7aef49f974a3" (UID: "d505d54d-e785-4f07-87c1-7aef49f974a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:25.971694 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.971638 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-istio-envoy" pod "d505d54d-e785-4f07-87c1-7aef49f974a3" (UID: "d505d54d-e785-4f07-87c1-7aef49f974a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:25.971694 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.971645 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d505d54d-e785-4f07-87c1-7aef49f974a3-istio-token" pod "d505d54d-e785-4f07-87c1-7aef49f974a3" (UID: "d505d54d-e785-4f07-87c1-7aef49f974a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:25.971694 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.971683 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d505d54d-e785-4f07-87c1-7aef49f974a3-kube-api-access-n2b4h" pod "d505d54d-e785-4f07-87c1-7aef49f974a3" (UID: "d505d54d-e785-4f07-87c1-7aef49f974a3"). InnerVolumeSpecName "kube-api-access-n2b4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:25.972812 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:25.972791 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d505d54d-e785-4f07-87c1-7aef49f974a3-istio-podinfo" pod "d505d54d-e785-4f07-87c1-7aef49f974a3" (UID: "d505d54d-e785-4f07-87c1-7aef49f974a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:55:26.061347 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:26.061282 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-workload-socket\") pod \"d505d54d-e785-4f07-87c1-7aef49f974a3\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " Nov 02 17:55:26.061530 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:26.061466 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-istio-data\") pod \"d505d54d-e785-4f07-87c1-7aef49f974a3\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " Nov 02 17:55:26.061530 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:26.061500 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-workload-socket" pod "d505d54d-e785-4f07-87c1-7aef49f974a3" (UID: "d505d54d-e785-4f07-87c1-7aef49f974a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:26.061658 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:26.061543 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d505d54d-e785-4f07-87c1-7aef49f974a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d505d54d-e785-4f07-87c1-7aef49f974a3-istiod-ca-cert\") pod \"d505d54d-e785-4f07-87c1-7aef49f974a3\" (UID: \"d505d54d-e785-4f07-87c1-7aef49f974a3\") " Nov 02 17:55:26.061797 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:26.061748 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-istio-data" pod "d505d54d-e785-4f07-87c1-7aef49f974a3" (UID: "d505d54d-e785-4f07-87c1-7aef49f974a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:26.061914 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:26.061892 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d505d54d-e785-4f07-87c1-7aef49f974a3-istiod-ca-cert" pod "d505d54d-e785-4f07-87c1-7aef49f974a3" (UID: "d505d54d-e785-4f07-87c1-7aef49f974a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:26.062389 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:26.062366 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:26.062529 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:26.062392 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:26.062529 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:26.062407 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:26.062529 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:26.062421 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:26.062529 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:26.062451 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d505d54d-e785-4f07-87c1-7aef49f974a3-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:26.062529 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:26.062466 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d505d54d-e785-4f07-87c1-7aef49f974a3-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:26.062529 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:26.062480 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2b4h\" (UniqueName: \"kubernetes.io/projected/d505d54d-e785-4f07-87c1-7aef49f974a3-kube-api-access-n2b4h\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:26.062529 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:26.062494 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d505d54d-e785-4f07-87c1-7aef49f974a3-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:26.062529 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:26.062506 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d505d54d-e785-4f07-87c1-7aef49f974a3-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:26.285403 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:55:26.285372 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19632b35_0e41_4d90_89c3_ced789733fbc.slice/crio-eefc2e420fee287b8e71cc04e1ea19117660d07bf99a7eada1eb077dfe4583ab WatchSource:0}: Error finding container eefc2e420fee287b8e71cc04e1ea19117660d07bf99a7eada1eb077dfe4583ab: Status 404 returned error can't find the container with id eefc2e420fee287b8e71cc04e1ea19117660d07bf99a7eada1eb077dfe4583ab Nov 02 17:55:26.286584 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:26.286561 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs"] Nov 02 17:55:26.288793 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:26.288758 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:55:26.288872 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:26.288831 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:55:26.288872 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:26.288860 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:55:26.723603 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:26.723518 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" Nov 02 17:55:26.723603 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:26.723560 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55" event={"ID":"d505d54d-e785-4f07-87c1-7aef49f974a3","Type":"ContainerDied","Data":"7bf96e026ea64cb01a811a8608e4f3539c25fa7cc0e5055c19100a1dc1f21324"} Nov 02 17:55:26.723840 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:26.723637 4471 scope.go:122] "RemoveContainer" containerID="be9bb81b215f788abfab429f970bb3c8b80255b51a78eab8169717feee0c5673" Nov 02 17:55:26.725702 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:26.725674 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" event={"ID":"19632b35-0e41-4d90-89c3-ced789733fbc","Type":"ContainerStarted","Data":"f9796c87e49aaa1a9c983d5d0155d4d5208bc74f1f4deef29a67a86286af22b7"} Nov 02 17:55:26.725854 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:26.725721 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" event={"ID":"19632b35-0e41-4d90-89c3-ced789733fbc","Type":"ContainerStarted","Data":"eefc2e420fee287b8e71cc04e1ea19117660d07bf99a7eada1eb077dfe4583ab"} Nov 02 17:55:26.781591 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:26.781551 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55"] Nov 02 17:55:26.794915 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:26.794889 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-h7v55"] Nov 02 17:55:26.843185 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:26.843127 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" podStartSLOduration=1.843113631 podStartE2EDuration="1.843113631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:26.841247067 +0000 UTC m=+1544.536602539" watchObservedRunningTime="2026-11-02 17:55:26.843113631 +0000 UTC m=+1544.538469105" Nov 02 17:55:26.964337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:26.963394 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:55:27.964738 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:27.964703 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:55:28.442374 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:28.442342 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d505d54d-e785-4f07-87c1-7aef49f974a3" path="/var/lib/kubelet/pods/d505d54d-e785-4f07-87c1-7aef49f974a3/volumes" Nov 02 17:55:28.741836 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:28.741759 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:55:28.742802 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:28.742780 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:55:31.271813 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:31.271781 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bb6d2e-5b8dd8bc9-pfhk7"] Nov 02 17:55:31.272622 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:31.272601 4471 cpu_manager.go:414] "containerMap: removing container" podUID="d505d54d-e785-4f07-87c1-7aef49f974a3" containerName="istio-proxy" Nov 02 17:55:31.272709 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:31.272624 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d505d54d-e785-4f07-87c1-7aef49f974a3" containerName="istio-proxy" Nov 02 17:55:31.272801 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:31.272791 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="d505d54d-e785-4f07-87c1-7aef49f974a3" containerName="istio-proxy" Nov 02 17:55:31.278896 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:31.278874 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb6d2e-5b8dd8bc9-pfhk7" Nov 02 17:55:31.304139 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:31.304113 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-bb6d2e\"" Nov 02 17:55:31.306642 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:31.306616 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bb6d2e-5b8dd8bc9-pfhk7"] Nov 02 17:55:31.315346 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:31.314172 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-bb6d2e-dockercfg-svwzn\"" Nov 02 17:55:31.368122 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:31.368089 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bb6d2e-5b8dd8bc9-pfhk7\" (UID: \"6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb6d2e-5b8dd8bc9-pfhk7" Nov 02 17:55:31.368264 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:31.368240 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zftlr\" (UniqueName: \"kubernetes.io/projected/6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8-kube-api-access-zftlr\") pod \"payload-pre-processing-e2e-inf-a-bb6d2e-5b8dd8bc9-pfhk7\" (UID: \"6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb6d2e-5b8dd8bc9-pfhk7" Nov 02 17:55:31.477329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:31.475046 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bb6d2e-5b8dd8bc9-pfhk7\" (UID: \"6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb6d2e-5b8dd8bc9-pfhk7" Nov 02 17:55:31.477329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:31.475198 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zftlr\" (UniqueName: \"kubernetes.io/projected/6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8-kube-api-access-zftlr\") pod \"payload-pre-processing-e2e-inf-a-bb6d2e-5b8dd8bc9-pfhk7\" (UID: \"6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb6d2e-5b8dd8bc9-pfhk7" Nov 02 17:55:31.477329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:31.476667 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bb6d2e-5b8dd8bc9-pfhk7\" (UID: \"6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb6d2e-5b8dd8bc9-pfhk7" Nov 02 17:55:31.553372 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:31.541381 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zftlr\" (UniqueName: \"kubernetes.io/projected/6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8-kube-api-access-zftlr\") pod \"payload-pre-processing-e2e-inf-a-bb6d2e-5b8dd8bc9-pfhk7\" (UID: \"6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb6d2e-5b8dd8bc9-pfhk7" Nov 02 17:55:31.605968 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:31.605146 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb6d2e-5b8dd8bc9-pfhk7" Nov 02 17:55:32.531681 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:55:32.531652 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3ff900_7ee5_40ce_8ba8_ca08e3f666f8.slice/crio-6568a670c694f6f7ebb4f927ed9b2cdfd9c9e1afc868264ad5c47869ab8cb152 WatchSource:0}: Error finding container 6568a670c694f6f7ebb4f927ed9b2cdfd9c9e1afc868264ad5c47869ab8cb152: Status 404 returned error can't find the container with id 6568a670c694f6f7ebb4f927ed9b2cdfd9c9e1afc868264ad5c47869ab8cb152 Nov 02 17:55:32.534250 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:32.534221 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bb6d2e-5b8dd8bc9-pfhk7"] Nov 02 17:55:32.831085 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:32.830988 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb6d2e-5b8dd8bc9-pfhk7" event={"ID":"6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8","Type":"ContainerStarted","Data":"c5bdf79f9882a21a80c93b7bea365f5c57b61be10def4c341913546c12a74db3"} Nov 02 17:55:32.831085 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:32.831060 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb6d2e-5b8dd8bc9-pfhk7" Nov 02 17:55:32.831085 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:32.831082 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb6d2e-5b8dd8bc9-pfhk7" event={"ID":"6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8","Type":"ContainerStarted","Data":"6568a670c694f6f7ebb4f927ed9b2cdfd9c9e1afc868264ad5c47869ab8cb152"} Nov 02 17:55:32.881940 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:32.881883 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb6d2e-5b8dd8bc9-pfhk7" podStartSLOduration=1.881864313 podStartE2EDuration="1.881864313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:32.881196388 +0000 UTC m=+1550.576551861" watchObservedRunningTime="2026-11-02 17:55:32.881864313 +0000 UTC m=+1550.577219793" Nov 02 17:55:34.403222 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:34.403195 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a209f-789b5bc59f-lgg4z" Nov 02 17:55:39.267433 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:55:39.267358 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:55:43.861997 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:43.861966 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb6d2e-5b8dd8bc9-pfhk7" Nov 02 17:55:48.517423 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:48.517382 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7"] Nov 02 17:55:48.517914 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:48.517636 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" podUID="9303eeae-3eca-4a5e-b2b2-c7671ecc41cb" containerName="istio-proxy" containerID="cri-o://aae29c16df8b2851c59533259ff74bf1c97cd4f10dac9ef546f36abf7e79172d" gracePeriod=30 Nov 02 17:55:53.820414 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:53.820387 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:55:53.879759 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:53.879728 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istio-envoy\") pod \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " Nov 02 17:55:53.879912 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:53.879886 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istio-podinfo\") pod \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " Nov 02 17:55:53.879955 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:53.879943 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istio-token\" (UniqueName: \"kubernetes.io/projected/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istio-token\") pod \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " Nov 02 17:55:53.879991 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:53.879979 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istiod-ca-cert\") pod \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " Nov 02 17:55:53.880039 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:53.880011 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-workload-socket\") pod \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " Nov 02 17:55:53.880155 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:53.880137 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-credential-socket\") pod \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " Nov 02 17:55:53.880228 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:53.880217 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-kube-api-access-hfn4x\" (UniqueName: \"kubernetes.io/projected/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-kube-api-access-hfn4x\") pod \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " Nov 02 17:55:53.880368 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:53.880347 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istio-data\") pod \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " Nov 02 17:55:53.880481 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:53.880388 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-workload-certs\") pod \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\" (UID: \"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb\") " Nov 02 17:55:53.881500 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:53.880915 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-credential-socket" pod "9303eeae-3eca-4a5e-b2b2-c7671ecc41cb" (UID: "9303eeae-3eca-4a5e-b2b2-c7671ecc41cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:53.881500 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:53.881135 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-workload-certs" pod "9303eeae-3eca-4a5e-b2b2-c7671ecc41cb" (UID: "9303eeae-3eca-4a5e-b2b2-c7671ecc41cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:53.881500 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:53.881150 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-workload-socket" pod "9303eeae-3eca-4a5e-b2b2-c7671ecc41cb" (UID: "9303eeae-3eca-4a5e-b2b2-c7671ecc41cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:53.881500 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:53.881338 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istio-data" pod "9303eeae-3eca-4a5e-b2b2-c7671ecc41cb" (UID: "9303eeae-3eca-4a5e-b2b2-c7671ecc41cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:53.881500 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:53.881432 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istiod-ca-cert" pod "9303eeae-3eca-4a5e-b2b2-c7671ecc41cb" (UID: "9303eeae-3eca-4a5e-b2b2-c7671ecc41cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:53.881846 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:53.881689 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:53.881846 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:53.881707 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:53.881846 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:53.881719 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:53.881846 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:53.881733 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:53.881846 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:53.881745 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:53.882595 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:53.882568 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istio-podinfo" pod "9303eeae-3eca-4a5e-b2b2-c7671ecc41cb" (UID: "9303eeae-3eca-4a5e-b2b2-c7671ecc41cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:55:53.882595 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:53.882583 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istio-envoy" pod "9303eeae-3eca-4a5e-b2b2-c7671ecc41cb" (UID: "9303eeae-3eca-4a5e-b2b2-c7671ecc41cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:53.882755 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:53.882621 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istio-token" pod "9303eeae-3eca-4a5e-b2b2-c7671ecc41cb" (UID: "9303eeae-3eca-4a5e-b2b2-c7671ecc41cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:53.882755 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:53.882723 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-kube-api-access-hfn4x" pod "9303eeae-3eca-4a5e-b2b2-c7671ecc41cb" (UID: "9303eeae-3eca-4a5e-b2b2-c7671ecc41cb"). InnerVolumeSpecName "kube-api-access-hfn4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:53.983568 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:53.983532 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfn4x\" (UniqueName: \"kubernetes.io/projected/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-kube-api-access-hfn4x\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:53.983568 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:53.983565 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:53.983568 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:53.983575 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:53.983897 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:53.983585 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:55:54.250825 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:54.250723 4471 generic.go:356] "Generic (PLEG): container finished" podID="9303eeae-3eca-4a5e-b2b2-c7671ecc41cb" containerID="aae29c16df8b2851c59533259ff74bf1c97cd4f10dac9ef546f36abf7e79172d" exitCode=0 Nov 02 17:55:54.250825 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:54.250799 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" Nov 02 17:55:54.250825 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:54.250805 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" event={"ID":"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb","Type":"ContainerDied","Data":"aae29c16df8b2851c59533259ff74bf1c97cd4f10dac9ef546f36abf7e79172d"} Nov 02 17:55:54.251091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:54.250887 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7" event={"ID":"9303eeae-3eca-4a5e-b2b2-c7671ecc41cb","Type":"ContainerDied","Data":"5ca90e8154f855226f92cc4d9b027d5fca2dadf3ba94631369180487899cf31b"} Nov 02 17:55:54.251091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:54.250918 4471 scope.go:122] "RemoveContainer" containerID="aae29c16df8b2851c59533259ff74bf1c97cd4f10dac9ef546f36abf7e79172d" Nov 02 17:55:54.263678 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:54.263658 4471 scope.go:122] "RemoveContainer" containerID="aae29c16df8b2851c59533259ff74bf1c97cd4f10dac9ef546f36abf7e79172d" Nov 02 17:55:54.263957 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:55:54.263938 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aae29c16df8b2851c59533259ff74bf1c97cd4f10dac9ef546f36abf7e79172d\": container with ID starting with aae29c16df8b2851c59533259ff74bf1c97cd4f10dac9ef546f36abf7e79172d not found: ID does not exist" containerID="aae29c16df8b2851c59533259ff74bf1c97cd4f10dac9ef546f36abf7e79172d" Nov 02 17:55:54.264025 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:54.263969 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aae29c16df8b2851c59533259ff74bf1c97cd4f10dac9ef546f36abf7e79172d"} err="failed to get container status \"aae29c16df8b2851c59533259ff74bf1c97cd4f10dac9ef546f36abf7e79172d\": rpc error: code = NotFound desc = could not find container \"aae29c16df8b2851c59533259ff74bf1c97cd4f10dac9ef546f36abf7e79172d\": container with ID starting with aae29c16df8b2851c59533259ff74bf1c97cd4f10dac9ef546f36abf7e79172d not found: ID does not exist" Nov 02 17:55:54.308727 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:54.308696 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7"] Nov 02 17:55:54.321198 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:54.321173 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-6slb7"] Nov 02 17:55:54.444476 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:55:54.444437 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9303eeae-3eca-4a5e-b2b2-c7671ecc41cb" path="/var/lib/kubelet/pods/9303eeae-3eca-4a5e-b2b2-c7671ecc41cb/volumes" Nov 02 17:56:08.905291 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:08.905086 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46"] Nov 02 17:56:08.909842 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:08.905415 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" podUID="c15b9c7c-b525-4e27-9b8c-5267adedd8b2" containerName="istio-proxy" containerID="cri-o://65e4a9347ffa2b1b189b8a44f59369f6f01d2c9c35125648fbeb6dcc0a461e22" gracePeriod=30 Nov 02 17:56:08.915001 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:08.914974 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8"] Nov 02 17:56:08.915236 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:08.915193 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" podUID="558a5960-22f7-4b6c-a71f-02e532e12576" containerName="istio-proxy" containerID="cri-o://ed54ab3565c8c5c44023431ced29501851e0290fd81a7dfbc90b10a4676b1450" gracePeriod=30 Nov 02 17:56:08.923183 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:08.923155 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z"] Nov 02 17:56:08.923376 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:08.923356 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" podUID="941519c9-0872-4d7c-a5b7-1045b696968a" containerName="istio-proxy" containerID="cri-o://8a1b79bcb54e620302c53dbb71b74f1237e86b76fde13be3fcf19b1343999716" gracePeriod=30 Nov 02 17:56:09.294635 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:56:09.294592 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:56:13.642658 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:13.642617 4471 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 17:56:13.643142 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:13.642690 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" podUID="941519c9-0872-4d7c-a5b7-1045b696968a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:56:13.706540 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:13.706503 4471 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 17:56:13.706807 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:13.706777 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" podUID="558a5960-22f7-4b6c-a71f-02e532e12576" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:56:14.317031 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.316997 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:56:14.322296 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.322258 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:56:14.388022 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.387991 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:56:14.458551 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.458464 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-workload-socket\") pod \"558a5960-22f7-4b6c-a71f-02e532e12576\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " Nov 02 17:56:14.458551 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.458516 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/558a5960-22f7-4b6c-a71f-02e532e12576-istio-token\" (UniqueName: \"kubernetes.io/projected/558a5960-22f7-4b6c-a71f-02e532e12576-istio-token\") pod \"558a5960-22f7-4b6c-a71f-02e532e12576\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " Nov 02 17:56:14.459255 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.458573 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/558a5960-22f7-4b6c-a71f-02e532e12576-kube-api-access-vf4dz\" (UniqueName: \"kubernetes.io/projected/558a5960-22f7-4b6c-a71f-02e532e12576-kube-api-access-vf4dz\") pod \"558a5960-22f7-4b6c-a71f-02e532e12576\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " Nov 02 17:56:14.459255 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.458622 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-workload-certs\") pod \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " Nov 02 17:56:14.459255 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.458652 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istio-data\") pod \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " Nov 02 17:56:14.459255 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.458674 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-credential-socket\") pod \"558a5960-22f7-4b6c-a71f-02e532e12576\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " Nov 02 17:56:14.459255 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.458754 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istio-envoy\") pod \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " Nov 02 17:56:14.459255 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.458787 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istio-token\" (UniqueName: \"kubernetes.io/projected/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istio-token\") pod \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " Nov 02 17:56:14.459255 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.458814 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-workload-certs\") pod \"558a5960-22f7-4b6c-a71f-02e532e12576\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " Nov 02 17:56:14.459255 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.458846 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-istio-envoy\") pod \"558a5960-22f7-4b6c-a71f-02e532e12576\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " Nov 02 17:56:14.459255 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.458935 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/558a5960-22f7-4b6c-a71f-02e532e12576-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558a5960-22f7-4b6c-a71f-02e532e12576-istiod-ca-cert\") pod \"558a5960-22f7-4b6c-a71f-02e532e12576\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " Nov 02 17:56:14.459255 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.458964 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-credential-socket\") pod \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " Nov 02 17:56:14.459255 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.459003 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-kube-api-access-wl5zf\" (UniqueName: \"kubernetes.io/projected/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-kube-api-access-wl5zf\") pod \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " Nov 02 17:56:14.459255 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.459068 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/558a5960-22f7-4b6c-a71f-02e532e12576-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558a5960-22f7-4b6c-a71f-02e532e12576-istio-podinfo\") pod \"558a5960-22f7-4b6c-a71f-02e532e12576\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " Nov 02 17:56:14.459255 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.459102 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istiod-ca-cert\") pod \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " Nov 02 17:56:14.459255 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.459146 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-istio-data\") pod \"558a5960-22f7-4b6c-a71f-02e532e12576\" (UID: \"558a5960-22f7-4b6c-a71f-02e532e12576\") " Nov 02 17:56:14.460914 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.459353 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istio-podinfo\") pod \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " Nov 02 17:56:14.460914 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.459385 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-workload-socket\") pod \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\" (UID: \"c15b9c7c-b525-4e27-9b8c-5267adedd8b2\") " Nov 02 17:56:14.460914 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.460525 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-workload-socket" pod "c15b9c7c-b525-4e27-9b8c-5267adedd8b2" (UID: "c15b9c7c-b525-4e27-9b8c-5267adedd8b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:14.460914 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.460750 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-istio-data" pod "558a5960-22f7-4b6c-a71f-02e532e12576" (UID: "558a5960-22f7-4b6c-a71f-02e532e12576"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:14.460914 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.460864 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istiod-ca-cert" pod "c15b9c7c-b525-4e27-9b8c-5267adedd8b2" (UID: "c15b9c7c-b525-4e27-9b8c-5267adedd8b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:56:14.461907 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.461463 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-workload-certs" pod "c15b9c7c-b525-4e27-9b8c-5267adedd8b2" (UID: "c15b9c7c-b525-4e27-9b8c-5267adedd8b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:14.462982 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.462918 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-workload-certs" pod "558a5960-22f7-4b6c-a71f-02e532e12576" (UID: "558a5960-22f7-4b6c-a71f-02e532e12576"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:14.464450 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.463923 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/558a5960-22f7-4b6c-a71f-02e532e12576-istiod-ca-cert" pod "558a5960-22f7-4b6c-a71f-02e532e12576" (UID: "558a5960-22f7-4b6c-a71f-02e532e12576"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:56:14.465154 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.464845 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-workload-socket" pod "558a5960-22f7-4b6c-a71f-02e532e12576" (UID: "558a5960-22f7-4b6c-a71f-02e532e12576"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:14.465675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.465293 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-credential-socket" pod "c15b9c7c-b525-4e27-9b8c-5267adedd8b2" (UID: "c15b9c7c-b525-4e27-9b8c-5267adedd8b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:14.466321 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.465882 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-credential-socket" pod "558a5960-22f7-4b6c-a71f-02e532e12576" (UID: "558a5960-22f7-4b6c-a71f-02e532e12576"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:14.467492 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.467451 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istio-data" pod "c15b9c7c-b525-4e27-9b8c-5267adedd8b2" (UID: "c15b9c7c-b525-4e27-9b8c-5267adedd8b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:14.468170 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.468131 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istio-envoy" pod "c15b9c7c-b525-4e27-9b8c-5267adedd8b2" (UID: "c15b9c7c-b525-4e27-9b8c-5267adedd8b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:14.472211 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.472185 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istio-podinfo" pod "c15b9c7c-b525-4e27-9b8c-5267adedd8b2" (UID: "c15b9c7c-b525-4e27-9b8c-5267adedd8b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:56:14.472341 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.472225 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/558a5960-22f7-4b6c-a71f-02e532e12576-kube-api-access-vf4dz" pod "558a5960-22f7-4b6c-a71f-02e532e12576" (UID: "558a5960-22f7-4b6c-a71f-02e532e12576"). InnerVolumeSpecName "kube-api-access-vf4dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:14.472469 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.472434 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/558a5960-22f7-4b6c-a71f-02e532e12576-istio-token" pod "558a5960-22f7-4b6c-a71f-02e532e12576" (UID: "558a5960-22f7-4b6c-a71f-02e532e12576"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:14.472597 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.472576 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-kube-api-access-wl5zf" pod "c15b9c7c-b525-4e27-9b8c-5267adedd8b2" (UID: "c15b9c7c-b525-4e27-9b8c-5267adedd8b2"). InnerVolumeSpecName "kube-api-access-wl5zf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:14.472711 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.472671 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/558a5960-22f7-4b6c-a71f-02e532e12576-istio-podinfo" pod "558a5960-22f7-4b6c-a71f-02e532e12576" (UID: "558a5960-22f7-4b6c-a71f-02e532e12576"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:56:14.473886 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.473858 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-istio-envoy" pod "558a5960-22f7-4b6c-a71f-02e532e12576" (UID: "558a5960-22f7-4b6c-a71f-02e532e12576"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:14.480504 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.479256 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istio-token" pod "c15b9c7c-b525-4e27-9b8c-5267adedd8b2" (UID: "c15b9c7c-b525-4e27-9b8c-5267adedd8b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:14.561277 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.561246 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-istio-data\") pod \"941519c9-0872-4d7c-a5b7-1045b696968a\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " Nov 02 17:56:14.561452 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.561366 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/941519c9-0872-4d7c-a5b7-1045b696968a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/941519c9-0872-4d7c-a5b7-1045b696968a-istiod-ca-cert\") pod \"941519c9-0872-4d7c-a5b7-1045b696968a\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " Nov 02 17:56:14.561452 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.561394 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-workload-socket\") pod \"941519c9-0872-4d7c-a5b7-1045b696968a\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " Nov 02 17:56:14.561452 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.561427 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-istio-envoy\") pod \"941519c9-0872-4d7c-a5b7-1045b696968a\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " Nov 02 17:56:14.561568 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.561471 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/941519c9-0872-4d7c-a5b7-1045b696968a-istio-token\" (UniqueName: \"kubernetes.io/projected/941519c9-0872-4d7c-a5b7-1045b696968a-istio-token\") pod \"941519c9-0872-4d7c-a5b7-1045b696968a\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " Nov 02 17:56:14.561568 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.561497 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/941519c9-0872-4d7c-a5b7-1045b696968a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/941519c9-0872-4d7c-a5b7-1045b696968a-istio-podinfo\") pod \"941519c9-0872-4d7c-a5b7-1045b696968a\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " Nov 02 17:56:14.561568 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.561540 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/941519c9-0872-4d7c-a5b7-1045b696968a-kube-api-access-czr8r\" (UniqueName: \"kubernetes.io/projected/941519c9-0872-4d7c-a5b7-1045b696968a-kube-api-access-czr8r\") pod \"941519c9-0872-4d7c-a5b7-1045b696968a\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " Nov 02 17:56:14.561568 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.561539 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-istio-data" pod "941519c9-0872-4d7c-a5b7-1045b696968a" (UID: "941519c9-0872-4d7c-a5b7-1045b696968a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:14.561774 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.561577 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-workload-certs\") pod \"941519c9-0872-4d7c-a5b7-1045b696968a\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " Nov 02 17:56:14.561774 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.561624 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-credential-socket\") pod \"941519c9-0872-4d7c-a5b7-1045b696968a\" (UID: \"941519c9-0872-4d7c-a5b7-1045b696968a\") " Nov 02 17:56:14.561774 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.561673 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-workload-socket" pod "941519c9-0872-4d7c-a5b7-1045b696968a" (UID: "941519c9-0872-4d7c-a5b7-1045b696968a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:14.561938 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.561836 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/941519c9-0872-4d7c-a5b7-1045b696968a-istiod-ca-cert" pod "941519c9-0872-4d7c-a5b7-1045b696968a" (UID: "941519c9-0872-4d7c-a5b7-1045b696968a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:56:14.562000 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.561925 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-workload-certs" pod "941519c9-0872-4d7c-a5b7-1045b696968a" (UID: "941519c9-0872-4d7c-a5b7-1045b696968a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:14.562133 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.562111 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-credential-socket" pod "941519c9-0872-4d7c-a5b7-1045b696968a" (UID: "941519c9-0872-4d7c-a5b7-1045b696968a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:14.562795 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.562543 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/558a5960-22f7-4b6c-a71f-02e532e12576-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.562795 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.562569 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf4dz\" (UniqueName: \"kubernetes.io/projected/558a5960-22f7-4b6c-a71f-02e532e12576-kube-api-access-vf4dz\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.562795 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.562585 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.562795 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.562600 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.562795 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.562613 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.562795 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.562626 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.562795 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.562640 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.562795 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.562655 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.562795 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.562672 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.562795 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.562686 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.562795 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.562700 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.562795 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.562713 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558a5960-22f7-4b6c-a71f-02e532e12576-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.562795 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.562728 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.562795 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.562743 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl5zf\" (UniqueName: \"kubernetes.io/projected/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-kube-api-access-wl5zf\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.562795 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.562758 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558a5960-22f7-4b6c-a71f-02e532e12576-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.562795 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.562773 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.562795 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.562784 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.562795 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.562795 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.562795 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.562807 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.563480 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.562819 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15b9c7c-b525-4e27-9b8c-5267adedd8b2-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.563480 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.562830 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/941519c9-0872-4d7c-a5b7-1045b696968a-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.563480 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.562844 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.563480 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.562855 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558a5960-22f7-4b6c-a71f-02e532e12576-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.563881 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.563851 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-istio-envoy" pod "941519c9-0872-4d7c-a5b7-1045b696968a" (UID: "941519c9-0872-4d7c-a5b7-1045b696968a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:14.563954 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.563909 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/941519c9-0872-4d7c-a5b7-1045b696968a-kube-api-access-czr8r" pod "941519c9-0872-4d7c-a5b7-1045b696968a" (UID: "941519c9-0872-4d7c-a5b7-1045b696968a"). InnerVolumeSpecName "kube-api-access-czr8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:14.564131 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.564117 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/941519c9-0872-4d7c-a5b7-1045b696968a-istio-podinfo" pod "941519c9-0872-4d7c-a5b7-1045b696968a" (UID: "941519c9-0872-4d7c-a5b7-1045b696968a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:56:14.564220 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.564201 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/941519c9-0872-4d7c-a5b7-1045b696968a-istio-token" pod "941519c9-0872-4d7c-a5b7-1045b696968a" (UID: "941519c9-0872-4d7c-a5b7-1045b696968a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:14.664492 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.664463 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/941519c9-0872-4d7c-a5b7-1045b696968a-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.664492 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.664489 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/941519c9-0872-4d7c-a5b7-1045b696968a-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.664492 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.664498 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/941519c9-0872-4d7c-a5b7-1045b696968a-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.665084 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.664506 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czr8r\" (UniqueName: \"kubernetes.io/projected/941519c9-0872-4d7c-a5b7-1045b696968a-kube-api-access-czr8r\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.672231 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.672202 4471 generic.go:356] "Generic (PLEG): container finished" podID="941519c9-0872-4d7c-a5b7-1045b696968a" containerID="8a1b79bcb54e620302c53dbb71b74f1237e86b76fde13be3fcf19b1343999716" exitCode=0 Nov 02 17:56:14.672374 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.672247 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" Nov 02 17:56:14.672374 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.672288 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" event={"ID":"941519c9-0872-4d7c-a5b7-1045b696968a","Type":"ContainerDied","Data":"8a1b79bcb54e620302c53dbb71b74f1237e86b76fde13be3fcf19b1343999716"} Nov 02 17:56:14.672374 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.672369 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z" event={"ID":"941519c9-0872-4d7c-a5b7-1045b696968a","Type":"ContainerDied","Data":"ef26a1d48ce2840a6f800b27299dbac3006351851c8db9f2c693d30145e96a21"} Nov 02 17:56:14.672516 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.672389 4471 scope.go:122] "RemoveContainer" containerID="8a1b79bcb54e620302c53dbb71b74f1237e86b76fde13be3fcf19b1343999716" Nov 02 17:56:14.674045 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.674009 4471 generic.go:356] "Generic (PLEG): container finished" podID="c15b9c7c-b525-4e27-9b8c-5267adedd8b2" containerID="65e4a9347ffa2b1b189b8a44f59369f6f01d2c9c35125648fbeb6dcc0a461e22" exitCode=0 Nov 02 17:56:14.674159 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.674060 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" Nov 02 17:56:14.674159 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.674058 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" event={"ID":"c15b9c7c-b525-4e27-9b8c-5267adedd8b2","Type":"ContainerDied","Data":"65e4a9347ffa2b1b189b8a44f59369f6f01d2c9c35125648fbeb6dcc0a461e22"} Nov 02 17:56:14.674271 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.674172 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46" event={"ID":"c15b9c7c-b525-4e27-9b8c-5267adedd8b2","Type":"ContainerDied","Data":"f63a8a1c1fcce8115888fc7aa832bf2ac9985472c73d764bb170707754b0e4a5"} Nov 02 17:56:14.675883 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.675853 4471 generic.go:356] "Generic (PLEG): container finished" podID="558a5960-22f7-4b6c-a71f-02e532e12576" containerID="ed54ab3565c8c5c44023431ced29501851e0290fd81a7dfbc90b10a4676b1450" exitCode=0 Nov 02 17:56:14.676002 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.675894 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" Nov 02 17:56:14.676002 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.675902 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" event={"ID":"558a5960-22f7-4b6c-a71f-02e532e12576","Type":"ContainerDied","Data":"ed54ab3565c8c5c44023431ced29501851e0290fd81a7dfbc90b10a4676b1450"} Nov 02 17:56:14.676002 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.675940 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8" event={"ID":"558a5960-22f7-4b6c-a71f-02e532e12576","Type":"ContainerDied","Data":"35abe4822ef8c243bab09b85d075c84046c2f973cc6e97149ca94615dc18e526"} Nov 02 17:56:14.684647 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.684623 4471 scope.go:122] "RemoveContainer" containerID="8a1b79bcb54e620302c53dbb71b74f1237e86b76fde13be3fcf19b1343999716" Nov 02 17:56:14.684899 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:56:14.684881 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a1b79bcb54e620302c53dbb71b74f1237e86b76fde13be3fcf19b1343999716\": container with ID starting with 8a1b79bcb54e620302c53dbb71b74f1237e86b76fde13be3fcf19b1343999716 not found: ID does not exist" containerID="8a1b79bcb54e620302c53dbb71b74f1237e86b76fde13be3fcf19b1343999716" Nov 02 17:56:14.685004 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.684909 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a1b79bcb54e620302c53dbb71b74f1237e86b76fde13be3fcf19b1343999716"} err="failed to get container status \"8a1b79bcb54e620302c53dbb71b74f1237e86b76fde13be3fcf19b1343999716\": rpc error: code = NotFound desc = could not find container \"8a1b79bcb54e620302c53dbb71b74f1237e86b76fde13be3fcf19b1343999716\": container with ID starting with 8a1b79bcb54e620302c53dbb71b74f1237e86b76fde13be3fcf19b1343999716 not found: ID does not exist" Nov 02 17:56:14.685004 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.684931 4471 scope.go:122] "RemoveContainer" containerID="65e4a9347ffa2b1b189b8a44f59369f6f01d2c9c35125648fbeb6dcc0a461e22" Nov 02 17:56:14.696100 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.696084 4471 scope.go:122] "RemoveContainer" containerID="65e4a9347ffa2b1b189b8a44f59369f6f01d2c9c35125648fbeb6dcc0a461e22" Nov 02 17:56:14.697878 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:56:14.697832 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65e4a9347ffa2b1b189b8a44f59369f6f01d2c9c35125648fbeb6dcc0a461e22\": container with ID starting with 65e4a9347ffa2b1b189b8a44f59369f6f01d2c9c35125648fbeb6dcc0a461e22 not found: ID does not exist" containerID="65e4a9347ffa2b1b189b8a44f59369f6f01d2c9c35125648fbeb6dcc0a461e22" Nov 02 17:56:14.697973 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.697883 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65e4a9347ffa2b1b189b8a44f59369f6f01d2c9c35125648fbeb6dcc0a461e22"} err="failed to get container status \"65e4a9347ffa2b1b189b8a44f59369f6f01d2c9c35125648fbeb6dcc0a461e22\": rpc error: code = NotFound desc = could not find container \"65e4a9347ffa2b1b189b8a44f59369f6f01d2c9c35125648fbeb6dcc0a461e22\": container with ID starting with 65e4a9347ffa2b1b189b8a44f59369f6f01d2c9c35125648fbeb6dcc0a461e22 not found: ID does not exist" Nov 02 17:56:14.697973 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.697900 4471 scope.go:122] "RemoveContainer" containerID="ed54ab3565c8c5c44023431ced29501851e0290fd81a7dfbc90b10a4676b1450" Nov 02 17:56:14.708549 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.708529 4471 scope.go:122] "RemoveContainer" containerID="ed54ab3565c8c5c44023431ced29501851e0290fd81a7dfbc90b10a4676b1450" Nov 02 17:56:14.708893 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:56:14.708838 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed54ab3565c8c5c44023431ced29501851e0290fd81a7dfbc90b10a4676b1450\": container with ID starting with ed54ab3565c8c5c44023431ced29501851e0290fd81a7dfbc90b10a4676b1450 not found: ID does not exist" containerID="ed54ab3565c8c5c44023431ced29501851e0290fd81a7dfbc90b10a4676b1450" Nov 02 17:56:14.708893 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.708865 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed54ab3565c8c5c44023431ced29501851e0290fd81a7dfbc90b10a4676b1450"} err="failed to get container status \"ed54ab3565c8c5c44023431ced29501851e0290fd81a7dfbc90b10a4676b1450\": rpc error: code = NotFound desc = could not find container \"ed54ab3565c8c5c44023431ced29501851e0290fd81a7dfbc90b10a4676b1450\": container with ID starting with ed54ab3565c8c5c44023431ced29501851e0290fd81a7dfbc90b10a4676b1450 not found: ID does not exist" Nov 02 17:56:14.728573 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.728547 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46"] Nov 02 17:56:14.740495 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.740476 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-tvv46"] Nov 02 17:56:14.767466 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.767442 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z"] Nov 02 17:56:14.773594 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.773570 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-fll6z"] Nov 02 17:56:14.811944 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.811918 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8"] Nov 02 17:56:14.818581 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:14.818555 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-hp4b8"] Nov 02 17:56:16.443054 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:16.443021 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="558a5960-22f7-4b6c-a71f-02e532e12576" path="/var/lib/kubelet/pods/558a5960-22f7-4b6c-a71f-02e532e12576/volumes" Nov 02 17:56:16.443645 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:16.443628 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="941519c9-0872-4d7c-a5b7-1045b696968a" path="/var/lib/kubelet/pods/941519c9-0872-4d7c-a5b7-1045b696968a/volumes" Nov 02 17:56:16.444117 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:16.444103 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c15b9c7c-b525-4e27-9b8c-5267adedd8b2" path="/var/lib/kubelet/pods/c15b9c7c-b525-4e27-9b8c-5267adedd8b2/volumes" Nov 02 17:56:19.305511 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:19.305472 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp"] Nov 02 17:56:19.306436 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:19.306396 4471 cpu_manager.go:414] "containerMap: removing container" podUID="9303eeae-3eca-4a5e-b2b2-c7671ecc41cb" containerName="istio-proxy" Nov 02 17:56:19.306436 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:19.306422 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9303eeae-3eca-4a5e-b2b2-c7671ecc41cb" containerName="istio-proxy" Nov 02 17:56:19.306654 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:19.306445 4471 cpu_manager.go:414] "containerMap: removing container" podUID="558a5960-22f7-4b6c-a71f-02e532e12576" containerName="istio-proxy" Nov 02 17:56:19.306654 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:19.306451 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="558a5960-22f7-4b6c-a71f-02e532e12576" containerName="istio-proxy" Nov 02 17:56:19.306654 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:19.306484 4471 cpu_manager.go:414] "containerMap: removing container" podUID="c15b9c7c-b525-4e27-9b8c-5267adedd8b2" containerName="istio-proxy" Nov 02 17:56:19.306654 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:19.306494 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c15b9c7c-b525-4e27-9b8c-5267adedd8b2" containerName="istio-proxy" Nov 02 17:56:19.306654 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:19.306512 4471 cpu_manager.go:414] "containerMap: removing container" podUID="941519c9-0872-4d7c-a5b7-1045b696968a" containerName="istio-proxy" Nov 02 17:56:19.306654 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:19.306518 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="941519c9-0872-4d7c-a5b7-1045b696968a" containerName="istio-proxy" Nov 02 17:56:19.306962 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:19.306680 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="c15b9c7c-b525-4e27-9b8c-5267adedd8b2" containerName="istio-proxy" Nov 02 17:56:19.306962 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:19.306697 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="941519c9-0872-4d7c-a5b7-1045b696968a" containerName="istio-proxy" Nov 02 17:56:19.306962 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:19.306716 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="9303eeae-3eca-4a5e-b2b2-c7671ecc41cb" containerName="istio-proxy" Nov 02 17:56:19.306962 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:19.306729 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="558a5960-22f7-4b6c-a71f-02e532e12576" containerName="istio-proxy" Nov 02 17:56:19.340335 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:19.327107 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp" Nov 02 17:56:19.386371 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:19.345607 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:56:19.386371 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:19.360362 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a29999-serving-cert\"" Nov 02 17:56:19.394442 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:19.389940 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp"] Nov 02 17:56:19.467330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:19.460480 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8627beff-73fc-4a24-a846-55d6a27ce9b7-maas-api-tls\") pod \"maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp\" (UID: \"8627beff-73fc-4a24-a846-55d6a27ce9b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp" Nov 02 17:56:19.467330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:19.460809 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8627beff-73fc-4a24-a846-55d6a27ce9b7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp\" (UID: \"8627beff-73fc-4a24-a846-55d6a27ce9b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp" Nov 02 17:56:19.467330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:19.460891 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8627beff-73fc-4a24-a846-55d6a27ce9b7-metrics-certs\") pod \"maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp\" (UID: \"8627beff-73fc-4a24-a846-55d6a27ce9b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp" Nov 02 17:56:19.467330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:19.460970 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qd8d\" (UniqueName: \"kubernetes.io/projected/8627beff-73fc-4a24-a846-55d6a27ce9b7-kube-api-access-4qd8d\") pod \"maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp\" (UID: \"8627beff-73fc-4a24-a846-55d6a27ce9b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp" Nov 02 17:56:19.611440 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:19.609823 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8627beff-73fc-4a24-a846-55d6a27ce9b7-metrics-certs\") pod \"maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp\" (UID: \"8627beff-73fc-4a24-a846-55d6a27ce9b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp" Nov 02 17:56:19.611440 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:19.609933 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qd8d\" (UniqueName: \"kubernetes.io/projected/8627beff-73fc-4a24-a846-55d6a27ce9b7-kube-api-access-4qd8d\") pod \"maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp\" (UID: \"8627beff-73fc-4a24-a846-55d6a27ce9b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp" Nov 02 17:56:19.611440 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:19.610007 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8627beff-73fc-4a24-a846-55d6a27ce9b7-maas-api-tls\") pod \"maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp\" (UID: \"8627beff-73fc-4a24-a846-55d6a27ce9b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp" Nov 02 17:56:19.611440 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:19.610364 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8627beff-73fc-4a24-a846-55d6a27ce9b7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp\" (UID: \"8627beff-73fc-4a24-a846-55d6a27ce9b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp" Nov 02 17:56:19.612549 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:19.611861 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8627beff-73fc-4a24-a846-55d6a27ce9b7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp\" (UID: \"8627beff-73fc-4a24-a846-55d6a27ce9b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp" Nov 02 17:56:19.622375 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:56:19.622344 4471 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-a29999-serving-cert: secret "maas-api-e2e-inf-a-a29999-serving-cert" not found Nov 02 17:56:19.624404 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:56:19.622647 4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8627beff-73fc-4a24-a846-55d6a27ce9b7-maas-api-tls podName:8627beff-73fc-4a24-a846-55d6a27ce9b7 nodeName:}" failed. No retries permitted until 2026-11-02 17:56:20.122621121 +0000 UTC m=+1597.817976575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8627beff-73fc-4a24-a846-55d6a27ce9b7-maas-api-tls") pod "maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp" (UID: "8627beff-73fc-4a24-a846-55d6a27ce9b7") : secret "maas-api-e2e-inf-a-a29999-serving-cert" not found Nov 02 17:56:19.652499 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:19.645792 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8627beff-73fc-4a24-a846-55d6a27ce9b7-metrics-certs\") pod \"maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp\" (UID: \"8627beff-73fc-4a24-a846-55d6a27ce9b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp" Nov 02 17:56:19.778331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:19.778233 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qd8d\" (UniqueName: \"kubernetes.io/projected/8627beff-73fc-4a24-a846-55d6a27ce9b7-kube-api-access-4qd8d\") pod \"maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp\" (UID: \"8627beff-73fc-4a24-a846-55d6a27ce9b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp" Nov 02 17:56:20.141680 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:20.141642 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8627beff-73fc-4a24-a846-55d6a27ce9b7-maas-api-tls\") pod \"maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp\" (UID: \"8627beff-73fc-4a24-a846-55d6a27ce9b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp" Nov 02 17:56:20.141880 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:56:20.141822 4471 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-a29999-serving-cert: secret "maas-api-e2e-inf-a-a29999-serving-cert" not found Nov 02 17:56:20.141926 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:56:20.141892 4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8627beff-73fc-4a24-a846-55d6a27ce9b7-maas-api-tls podName:8627beff-73fc-4a24-a846-55d6a27ce9b7 nodeName:}" failed. No retries permitted until 2026-11-02 17:56:21.141877667 +0000 UTC m=+1598.837233119 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8627beff-73fc-4a24-a846-55d6a27ce9b7-maas-api-tls") pod "maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp" (UID: "8627beff-73fc-4a24-a846-55d6a27ce9b7") : secret "maas-api-e2e-inf-a-a29999-serving-cert" not found Nov 02 17:56:20.401804 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:20.400686 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-227833-584bc86fbc-j9s47"] Nov 02 17:56:20.425487 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:20.425458 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-227833-55fc9bbf4c-vkgwf"] Nov 02 17:56:21.226333 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:21.223471 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8627beff-73fc-4a24-a846-55d6a27ce9b7-maas-api-tls\") pod \"maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp\" (UID: \"8627beff-73fc-4a24-a846-55d6a27ce9b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp" Nov 02 17:56:21.241326 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:21.236078 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8627beff-73fc-4a24-a846-55d6a27ce9b7-maas-api-tls\") pod \"maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp\" (UID: \"8627beff-73fc-4a24-a846-55d6a27ce9b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp" Nov 02 17:56:21.526330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:21.525837 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp" Nov 02 17:56:22.089470 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:22.089435 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp"] Nov 02 17:56:22.923360 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:22.921494 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp" event={"ID":"8627beff-73fc-4a24-a846-55d6a27ce9b7","Type":"ContainerStarted","Data":"af1cb59d1989dd264675a54dede81d2f12bfefe92a4a5aa15daee50960577b62"} Nov 02 17:56:22.923360 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:22.921575 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp" event={"ID":"8627beff-73fc-4a24-a846-55d6a27ce9b7","Type":"ContainerStarted","Data":"205c4967cb3ef0c96d330857e99cb2e782a8a79b2434443a7f284ab2ea3d5dfe"} Nov 02 17:56:22.923360 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:22.923045 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp" Nov 02 17:56:23.078385 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:23.077733 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp" podStartSLOduration=3.671997535 podStartE2EDuration="4.07771357s" totalImagesPullingTime="405.716035ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:22.993734038 +0000 UTC m=+1600.689089515" watchObservedRunningTime="2026-11-02 17:56:23.07771357 +0000 UTC m=+1600.773069050" Nov 02 17:56:23.896340 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:23.895411 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 02 17:56:24.090102 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:24.090057 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 02 17:56:24.214838 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:24.214729 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:56:25.417338 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:25.413878 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-227833-584bc86fbc-j9s47" podUID="9c40f8a7-55a8-479b-8305-68f236fd9749" containerName="payload-processing" containerID="cri-o://7086d0eca5abcc4af9a630c6123a1bb4e8ef8a986d9629fa49d9a71d0b0ab48a" gracePeriod=25 Nov 02 17:56:25.429160 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:25.428337 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-227833-55fc9bbf4c-vkgwf" podUID="9207e3dc-355e-468f-a9ba-7d4d83c1e8cd" containerName="payload-pre-processing" containerID="cri-o://aa296b89c4ce59399a1aa6a1261332052afd61138f6637bb7f743b2810faecda" gracePeriod=25 Nov 02 17:56:25.585549 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:56:25.584923 4471 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9207e3dc_355e_468f_a9ba_7d4d83c1e8cd.slice/crio-aa296b89c4ce59399a1aa6a1261332052afd61138f6637bb7f743b2810faecda.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c40f8a7_55a8_479b_8305_68f236fd9749.slice/crio-conmon-7086d0eca5abcc4af9a630c6123a1bb4e8ef8a986d9629fa49d9a71d0b0ab48a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c40f8a7_55a8_479b_8305_68f236fd9749.slice/crio-7086d0eca5abcc4af9a630c6123a1bb4e8ef8a986d9629fa49d9a71d0b0ab48a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9207e3dc_355e_468f_a9ba_7d4d83c1e8cd.slice/crio-conmon-aa296b89c4ce59399a1aa6a1261332052afd61138f6637bb7f743b2810faecda.scope\": RecentStats: unable to find data in memory cache]" Nov 02 17:56:25.585549 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:56:25.585361 4471 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c40f8a7_55a8_479b_8305_68f236fd9749.slice/crio-7086d0eca5abcc4af9a630c6123a1bb4e8ef8a986d9629fa49d9a71d0b0ab48a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9207e3dc_355e_468f_a9ba_7d4d83c1e8cd.slice/crio-conmon-aa296b89c4ce59399a1aa6a1261332052afd61138f6637bb7f743b2810faecda.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9207e3dc_355e_468f_a9ba_7d4d83c1e8cd.slice/crio-aa296b89c4ce59399a1aa6a1261332052afd61138f6637bb7f743b2810faecda.scope\": RecentStats: unable to find data in memory cache]" Nov 02 17:56:25.858855 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:25.858825 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-227833-55fc9bbf4c-vkgwf" Nov 02 17:56:25.909875 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:25.909852 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-227833-584bc86fbc-j9s47" Nov 02 17:56:25.944923 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:25.944885 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9207e3dc-355e-468f-a9ba-7d4d83c1e8cd-kube-api-access-4djkk\" (UniqueName: \"kubernetes.io/projected/9207e3dc-355e-468f-a9ba-7d4d83c1e8cd-kube-api-access-4djkk\") pod \"9207e3dc-355e-468f-a9ba-7d4d83c1e8cd\" (UID: \"9207e3dc-355e-468f-a9ba-7d4d83c1e8cd\") " Nov 02 17:56:25.945082 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:25.945065 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c40f8a7-55a8-479b-8305-68f236fd9749-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c40f8a7-55a8-479b-8305-68f236fd9749-plugins-config-volume\") pod \"9c40f8a7-55a8-479b-8305-68f236fd9749\" (UID: \"9c40f8a7-55a8-479b-8305-68f236fd9749\") " Nov 02 17:56:25.945133 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:25.945100 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c40f8a7-55a8-479b-8305-68f236fd9749-kube-api-access-s9gg7\" (UniqueName: \"kubernetes.io/projected/9c40f8a7-55a8-479b-8305-68f236fd9749-kube-api-access-s9gg7\") pod \"9c40f8a7-55a8-479b-8305-68f236fd9749\" (UID: \"9c40f8a7-55a8-479b-8305-68f236fd9749\") " Nov 02 17:56:25.945278 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:25.945255 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9207e3dc-355e-468f-a9ba-7d4d83c1e8cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9207e3dc-355e-468f-a9ba-7d4d83c1e8cd-plugins-config-volume\") pod \"9207e3dc-355e-468f-a9ba-7d4d83c1e8cd\" (UID: \"9207e3dc-355e-468f-a9ba-7d4d83c1e8cd\") " Nov 02 17:56:25.945529 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:25.945495 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c40f8a7-55a8-479b-8305-68f236fd9749-plugins-config-volume" pod "9c40f8a7-55a8-479b-8305-68f236fd9749" (UID: "9c40f8a7-55a8-479b-8305-68f236fd9749"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:56:25.945741 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:25.945711 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9207e3dc-355e-468f-a9ba-7d4d83c1e8cd-plugins-config-volume" pod "9207e3dc-355e-468f-a9ba-7d4d83c1e8cd" (UID: "9207e3dc-355e-468f-a9ba-7d4d83c1e8cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:56:25.946022 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:25.946003 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9207e3dc-355e-468f-a9ba-7d4d83c1e8cd-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:56:25.946090 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:25.946026 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c40f8a7-55a8-479b-8305-68f236fd9749-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:56:25.947104 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:25.947085 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9207e3dc-355e-468f-a9ba-7d4d83c1e8cd-kube-api-access-4djkk" pod "9207e3dc-355e-468f-a9ba-7d4d83c1e8cd" (UID: "9207e3dc-355e-468f-a9ba-7d4d83c1e8cd"). InnerVolumeSpecName "kube-api-access-4djkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:25.947166 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:25.947149 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c40f8a7-55a8-479b-8305-68f236fd9749-kube-api-access-s9gg7" pod "9c40f8a7-55a8-479b-8305-68f236fd9749" (UID: "9c40f8a7-55a8-479b-8305-68f236fd9749"). InnerVolumeSpecName "kube-api-access-s9gg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:26.049541 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:26.049491 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9gg7\" (UniqueName: \"kubernetes.io/projected/9c40f8a7-55a8-479b-8305-68f236fd9749-kube-api-access-s9gg7\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:56:26.049541 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:26.049525 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4djkk\" (UniqueName: \"kubernetes.io/projected/9207e3dc-355e-468f-a9ba-7d4d83c1e8cd-kube-api-access-4djkk\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:56:26.049820 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:26.049636 4471 generic.go:356] "Generic (PLEG): container finished" podID="9207e3dc-355e-468f-a9ba-7d4d83c1e8cd" containerID="aa296b89c4ce59399a1aa6a1261332052afd61138f6637bb7f743b2810faecda" exitCode=0 Nov 02 17:56:26.049913 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:26.049892 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-227833-55fc9bbf4c-vkgwf" Nov 02 17:56:26.049981 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:26.049956 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-227833-55fc9bbf4c-vkgwf" event={"ID":"9207e3dc-355e-468f-a9ba-7d4d83c1e8cd","Type":"ContainerDied","Data":"aa296b89c4ce59399a1aa6a1261332052afd61138f6637bb7f743b2810faecda"} Nov 02 17:56:26.050026 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:26.050010 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-227833-55fc9bbf4c-vkgwf" event={"ID":"9207e3dc-355e-468f-a9ba-7d4d83c1e8cd","Type":"ContainerDied","Data":"463095249fb36a5a9c57fa014aa4914689762e219b0523a7a3c0e2db72835f9a"} Nov 02 17:56:26.050072 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:26.050040 4471 scope.go:122] "RemoveContainer" containerID="aa296b89c4ce59399a1aa6a1261332052afd61138f6637bb7f743b2810faecda" Nov 02 17:56:26.060384 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:26.060354 4471 generic.go:356] "Generic (PLEG): container finished" podID="9c40f8a7-55a8-479b-8305-68f236fd9749" containerID="7086d0eca5abcc4af9a630c6123a1bb4e8ef8a986d9629fa49d9a71d0b0ab48a" exitCode=0 Nov 02 17:56:26.060522 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:26.060435 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-227833-584bc86fbc-j9s47" Nov 02 17:56:26.060592 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:26.060517 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-227833-584bc86fbc-j9s47" event={"ID":"9c40f8a7-55a8-479b-8305-68f236fd9749","Type":"ContainerDied","Data":"7086d0eca5abcc4af9a630c6123a1bb4e8ef8a986d9629fa49d9a71d0b0ab48a"} Nov 02 17:56:26.060592 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:26.060567 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-227833-584bc86fbc-j9s47" event={"ID":"9c40f8a7-55a8-479b-8305-68f236fd9749","Type":"ContainerDied","Data":"d8cc2eb756d0abfb1b4cdb6b3fa04232451327bb66178fc4d11eab66fe7dcb1b"} Nov 02 17:56:26.070804 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:26.070783 4471 scope.go:122] "RemoveContainer" containerID="aa296b89c4ce59399a1aa6a1261332052afd61138f6637bb7f743b2810faecda" Nov 02 17:56:26.071166 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:56:26.071145 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa296b89c4ce59399a1aa6a1261332052afd61138f6637bb7f743b2810faecda\": container with ID starting with aa296b89c4ce59399a1aa6a1261332052afd61138f6637bb7f743b2810faecda not found: ID does not exist" containerID="aa296b89c4ce59399a1aa6a1261332052afd61138f6637bb7f743b2810faecda" Nov 02 17:56:26.071245 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:26.071178 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa296b89c4ce59399a1aa6a1261332052afd61138f6637bb7f743b2810faecda"} err="failed to get container status \"aa296b89c4ce59399a1aa6a1261332052afd61138f6637bb7f743b2810faecda\": rpc error: code = NotFound desc = could not find container \"aa296b89c4ce59399a1aa6a1261332052afd61138f6637bb7f743b2810faecda\": container with ID starting with aa296b89c4ce59399a1aa6a1261332052afd61138f6637bb7f743b2810faecda not found: ID does not exist" Nov 02 17:56:26.071245 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:26.071202 4471 scope.go:122] "RemoveContainer" containerID="7086d0eca5abcc4af9a630c6123a1bb4e8ef8a986d9629fa49d9a71d0b0ab48a" Nov 02 17:56:26.084229 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:26.084209 4471 scope.go:122] "RemoveContainer" containerID="7086d0eca5abcc4af9a630c6123a1bb4e8ef8a986d9629fa49d9a71d0b0ab48a" Nov 02 17:56:26.084591 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:56:26.084568 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7086d0eca5abcc4af9a630c6123a1bb4e8ef8a986d9629fa49d9a71d0b0ab48a\": container with ID starting with 7086d0eca5abcc4af9a630c6123a1bb4e8ef8a986d9629fa49d9a71d0b0ab48a not found: ID does not exist" containerID="7086d0eca5abcc4af9a630c6123a1bb4e8ef8a986d9629fa49d9a71d0b0ab48a" Nov 02 17:56:26.084674 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:26.084601 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7086d0eca5abcc4af9a630c6123a1bb4e8ef8a986d9629fa49d9a71d0b0ab48a"} err="failed to get container status \"7086d0eca5abcc4af9a630c6123a1bb4e8ef8a986d9629fa49d9a71d0b0ab48a\": rpc error: code = NotFound desc = could not find container \"7086d0eca5abcc4af9a630c6123a1bb4e8ef8a986d9629fa49d9a71d0b0ab48a\": container with ID starting with 7086d0eca5abcc4af9a630c6123a1bb4e8ef8a986d9629fa49d9a71d0b0ab48a not found: ID does not exist" Nov 02 17:56:26.141959 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:26.141929 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-227833-55fc9bbf4c-vkgwf"] Nov 02 17:56:26.156197 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:26.156167 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-227833-55fc9bbf4c-vkgwf"] Nov 02 17:56:26.192077 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:26.192043 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-227833-584bc86fbc-j9s47"] Nov 02 17:56:26.198421 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:26.198391 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-227833-584bc86fbc-j9s47"] Nov 02 17:56:26.443545 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:26.443456 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9207e3dc-355e-468f-a9ba-7d4d83c1e8cd" path="/var/lib/kubelet/pods/9207e3dc-355e-468f-a9ba-7d4d83c1e8cd/volumes" Nov 02 17:56:26.443998 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:26.443982 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c40f8a7-55a8-479b-8305-68f236fd9749" path="/var/lib/kubelet/pods/9c40f8a7-55a8-479b-8305-68f236fd9749/volumes" Nov 02 17:56:30.039509 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.039476 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp" Nov 02 17:56:30.327158 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.327077 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv"] Nov 02 17:56:30.328240 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.328217 4471 cpu_manager.go:414] "containerMap: removing container" podUID="9c40f8a7-55a8-479b-8305-68f236fd9749" containerName="payload-processing" Nov 02 17:56:30.328240 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.328238 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c40f8a7-55a8-479b-8305-68f236fd9749" containerName="payload-processing" Nov 02 17:56:30.328439 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.328270 4471 cpu_manager.go:414] "containerMap: removing container" podUID="9207e3dc-355e-468f-a9ba-7d4d83c1e8cd" containerName="payload-pre-processing" Nov 02 17:56:30.328439 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.328278 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9207e3dc-355e-468f-a9ba-7d4d83c1e8cd" containerName="payload-pre-processing" Nov 02 17:56:30.328521 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.328479 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c40f8a7-55a8-479b-8305-68f236fd9749" containerName="payload-processing" Nov 02 17:56:30.328521 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.328497 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="9207e3dc-355e-468f-a9ba-7d4d83c1e8cd" containerName="payload-pre-processing" Nov 02 17:56:30.337421 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.334754 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:56:30.337421 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.336525 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv"] Nov 02 17:56:30.350986 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.350957 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b2fc9f-openshift-default-dockercfg-tqz7r\"" Nov 02 17:56:30.465327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.464683 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-workload-socket\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:56:30.465327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.464744 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istiod-ca-cert\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:56:30.465327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.464834 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istio-token\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:56:30.465327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.464893 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-workload-certs\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:56:30.465327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.464946 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istio-envoy\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:56:30.465327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.465005 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istio-data\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:56:30.465327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.465068 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istio-podinfo\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:56:30.465327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.465144 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-credential-socket\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:56:30.465327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.465339 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgvrz\" (UniqueName: \"kubernetes.io/projected/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-kube-api-access-sgvrz\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:56:30.586377 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.579382 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-workload-socket\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:56:30.586377 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.579432 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istiod-ca-cert\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:56:30.586377 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.579510 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istio-token\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:56:30.586377 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.579553 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-workload-certs\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:56:30.586377 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.579605 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istio-envoy\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:56:30.586377 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.579659 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istio-data\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:56:30.586377 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.579722 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istio-podinfo\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:56:30.586377 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.579794 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-credential-socket\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:56:30.586377 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.579965 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgvrz\" (UniqueName: \"kubernetes.io/projected/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-kube-api-access-sgvrz\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:56:30.586377 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.580790 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-workload-socket\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:56:30.586377 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.581475 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istiod-ca-cert\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:56:30.586377 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.581947 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-workload-certs\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:56:30.607944 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.605698 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-credential-socket\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:56:30.609215 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.609189 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istio-data\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:56:30.636334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.634895 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istio-podinfo\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:56:30.662334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.649604 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istio-envoy\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:56:30.662334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.650090 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istio-token\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:56:30.662334 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.650661 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgvrz\" (UniqueName: \"kubernetes.io/projected/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-kube-api-access-sgvrz\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:56:30.680023 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:30.667208 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:56:31.112818 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:31.112780 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv"] Nov 02 17:56:31.117861 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:56:31.117819 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c047abd_2c48_4b50_ba99_579d0f2cc3fd.slice/crio-9d4897367513b6428dc82f2b1589f1e36ceabe23269ab90f85c2edd20e5cf023 WatchSource:0}: Error finding container 9d4897367513b6428dc82f2b1589f1e36ceabe23269ab90f85c2edd20e5cf023: Status 404 returned error can't find the container with id 9d4897367513b6428dc82f2b1589f1e36ceabe23269ab90f85c2edd20e5cf023 Nov 02 17:56:31.122290 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:31.122252 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:56:31.122401 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:31.122366 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:56:31.122401 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:31.122396 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:56:32.119936 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:32.119892 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" event={"ID":"2c047abd-2c48-4b50-ba99-579d0f2cc3fd","Type":"ContainerStarted","Data":"15d32558f6a2658c05e1570f0d584c3a07008d0a460897a472f0f80216cfe32b"} Nov 02 17:56:32.120413 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:32.119956 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" event={"ID":"2c047abd-2c48-4b50-ba99-579d0f2cc3fd","Type":"ContainerStarted","Data":"9d4897367513b6428dc82f2b1589f1e36ceabe23269ab90f85c2edd20e5cf023"} Nov 02 17:56:32.182553 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:32.182501 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" podStartSLOduration=2.182486803 podStartE2EDuration="2.182486803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:32.178563276 +0000 UTC m=+1609.873918773" watchObservedRunningTime="2026-11-02 17:56:32.182486803 +0000 UTC m=+1609.877842275" Nov 02 17:56:32.668675 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:32.668638 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:56:32.674830 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:32.674804 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:56:33.129445 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:33.129409 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:56:33.130382 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:33.130357 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:56:36.734423 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:36.734381 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b2fc9f-5f789b6cdc-sdwdq"] Nov 02 17:56:36.738228 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:36.738209 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2fc9f-5f789b6cdc-sdwdq" Nov 02 17:56:36.774991 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:36.774960 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b2fc9f\"" Nov 02 17:56:36.775260 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:36.775187 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b2fc9f-dockercfg-mk9tp\"" Nov 02 17:56:36.823412 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:36.823377 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b2fc9f-6598c4cdc5-f5rrt"] Nov 02 17:56:36.885328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:36.843544 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03f6f5cc-5479-478e-8a95-b86c359d2935-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b2fc9f-5f789b6cdc-sdwdq\" (UID: \"03f6f5cc-5479-478e-8a95-b86c359d2935\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2fc9f-5f789b6cdc-sdwdq" Nov 02 17:56:36.885328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:36.843727 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcfn9\" (UniqueName: \"kubernetes.io/projected/03f6f5cc-5479-478e-8a95-b86c359d2935-kube-api-access-jcfn9\") pod \"payload-pre-processing-e2e-inf-b-b2fc9f-5f789b6cdc-sdwdq\" (UID: \"03f6f5cc-5479-478e-8a95-b86c359d2935\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2fc9f-5f789b6cdc-sdwdq" Nov 02 17:56:36.885328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:36.879138 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b2fc9f-5f789b6cdc-sdwdq"] Nov 02 17:56:36.885328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:36.879258 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b2fc9f-6598c4cdc5-f5rrt" Nov 02 17:56:36.979928 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:36.971807 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzlgd\" (UniqueName: \"kubernetes.io/projected/2bc68f8a-08a7-49b2-9564-f8041c6d96e0-kube-api-access-vzlgd\") pod \"payload-processing-e2e-inf-b-b2fc9f-6598c4cdc5-f5rrt\" (UID: \"2bc68f8a-08a7-49b2-9564-f8041c6d96e0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b2fc9f-6598c4cdc5-f5rrt" Nov 02 17:56:36.979928 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:36.971886 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03f6f5cc-5479-478e-8a95-b86c359d2935-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b2fc9f-5f789b6cdc-sdwdq\" (UID: \"03f6f5cc-5479-478e-8a95-b86c359d2935\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2fc9f-5f789b6cdc-sdwdq" Nov 02 17:56:36.979928 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:36.971971 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bc68f8a-08a7-49b2-9564-f8041c6d96e0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b2fc9f-6598c4cdc5-f5rrt\" (UID: \"2bc68f8a-08a7-49b2-9564-f8041c6d96e0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b2fc9f-6598c4cdc5-f5rrt" Nov 02 17:56:36.979928 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:36.972051 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcfn9\" (UniqueName: \"kubernetes.io/projected/03f6f5cc-5479-478e-8a95-b86c359d2935-kube-api-access-jcfn9\") pod \"payload-pre-processing-e2e-inf-b-b2fc9f-5f789b6cdc-sdwdq\" (UID: \"03f6f5cc-5479-478e-8a95-b86c359d2935\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2fc9f-5f789b6cdc-sdwdq" Nov 02 17:56:36.979928 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:36.974168 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03f6f5cc-5479-478e-8a95-b86c359d2935-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b2fc9f-5f789b6cdc-sdwdq\" (UID: \"03f6f5cc-5479-478e-8a95-b86c359d2935\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2fc9f-5f789b6cdc-sdwdq" Nov 02 17:56:37.041332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:37.038102 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcfn9\" (UniqueName: \"kubernetes.io/projected/03f6f5cc-5479-478e-8a95-b86c359d2935-kube-api-access-jcfn9\") pod \"payload-pre-processing-e2e-inf-b-b2fc9f-5f789b6cdc-sdwdq\" (UID: \"03f6f5cc-5479-478e-8a95-b86c359d2935\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2fc9f-5f789b6cdc-sdwdq" Nov 02 17:56:37.049968 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:37.041674 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b2fc9f-6598c4cdc5-f5rrt"] Nov 02 17:56:37.105387 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:37.105344 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzlgd\" (UniqueName: \"kubernetes.io/projected/2bc68f8a-08a7-49b2-9564-f8041c6d96e0-kube-api-access-vzlgd\") pod \"payload-processing-e2e-inf-b-b2fc9f-6598c4cdc5-f5rrt\" (UID: \"2bc68f8a-08a7-49b2-9564-f8041c6d96e0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b2fc9f-6598c4cdc5-f5rrt" Nov 02 17:56:37.107077 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:37.107046 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bc68f8a-08a7-49b2-9564-f8041c6d96e0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b2fc9f-6598c4cdc5-f5rrt\" (UID: \"2bc68f8a-08a7-49b2-9564-f8041c6d96e0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b2fc9f-6598c4cdc5-f5rrt" Nov 02 17:56:37.108489 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:37.106851 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2fc9f-5f789b6cdc-sdwdq" Nov 02 17:56:37.108925 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:37.108149 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bc68f8a-08a7-49b2-9564-f8041c6d96e0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b2fc9f-6598c4cdc5-f5rrt\" (UID: \"2bc68f8a-08a7-49b2-9564-f8041c6d96e0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b2fc9f-6598c4cdc5-f5rrt" Nov 02 17:56:37.135238 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:37.135166 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzlgd\" (UniqueName: \"kubernetes.io/projected/2bc68f8a-08a7-49b2-9564-f8041c6d96e0-kube-api-access-vzlgd\") pod \"payload-processing-e2e-inf-b-b2fc9f-6598c4cdc5-f5rrt\" (UID: \"2bc68f8a-08a7-49b2-9564-f8041c6d96e0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b2fc9f-6598c4cdc5-f5rrt" Nov 02 17:56:37.296490 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:37.287966 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b2fc9f-6598c4cdc5-f5rrt" Nov 02 17:56:37.464473 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:56:37.464400 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f6f5cc_5479_478e_8a95_b86c359d2935.slice/crio-92cc52e265cb7d44b0ac8028164bf92f25d6b1a5bb33acbfbb53e983e23e3bfa WatchSource:0}: Error finding container 92cc52e265cb7d44b0ac8028164bf92f25d6b1a5bb33acbfbb53e983e23e3bfa: Status 404 returned error can't find the container with id 92cc52e265cb7d44b0ac8028164bf92f25d6b1a5bb33acbfbb53e983e23e3bfa Nov 02 17:56:37.467827 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:37.467795 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b2fc9f-5f789b6cdc-sdwdq"] Nov 02 17:56:37.616716 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:56:37.616683 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc68f8a_08a7_49b2_9564_f8041c6d96e0.slice/crio-1d39d295c0f4d645e86b09473221d634d3d89caa1d113e61a5658e49e92676b9 WatchSource:0}: Error finding container 1d39d295c0f4d645e86b09473221d634d3d89caa1d113e61a5658e49e92676b9: Status 404 returned error can't find the container with id 1d39d295c0f4d645e86b09473221d634d3d89caa1d113e61a5658e49e92676b9 Nov 02 17:56:37.618697 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:37.618674 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b2fc9f-6598c4cdc5-f5rrt"] Nov 02 17:56:38.179537 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:38.179490 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b2fc9f-6598c4cdc5-f5rrt" event={"ID":"2bc68f8a-08a7-49b2-9564-f8041c6d96e0","Type":"ContainerStarted","Data":"4765540b783a51ac53326f08362de9a6f6cb511211083d2cdb0204211e7b5eae"} Nov 02 17:56:38.180009 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:38.179595 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b2fc9f-6598c4cdc5-f5rrt" Nov 02 17:56:38.180009 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:38.179623 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b2fc9f-6598c4cdc5-f5rrt" event={"ID":"2bc68f8a-08a7-49b2-9564-f8041c6d96e0","Type":"ContainerStarted","Data":"1d39d295c0f4d645e86b09473221d634d3d89caa1d113e61a5658e49e92676b9"} Nov 02 17:56:38.181583 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:38.181551 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2fc9f-5f789b6cdc-sdwdq" event={"ID":"03f6f5cc-5479-478e-8a95-b86c359d2935","Type":"ContainerStarted","Data":"3ea52fa1275e9053a773e2c2832774fad4d159f1b04614ff468d6013840a6f60"} Nov 02 17:56:38.181719 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:38.181600 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2fc9f-5f789b6cdc-sdwdq" event={"ID":"03f6f5cc-5479-478e-8a95-b86c359d2935","Type":"ContainerStarted","Data":"92cc52e265cb7d44b0ac8028164bf92f25d6b1a5bb33acbfbb53e983e23e3bfa"} Nov 02 17:56:38.181719 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:38.181698 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2fc9f-5f789b6cdc-sdwdq" Nov 02 17:56:38.226772 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:38.226714 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-b2fc9f-6598c4cdc5-f5rrt" podStartSLOduration=2.226695578 podStartE2EDuration="2.226695578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:38.225283887 +0000 UTC m=+1615.920639362" watchObservedRunningTime="2026-11-02 17:56:38.226695578 +0000 UTC m=+1615.922051057" Nov 02 17:56:38.271811 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:38.271753 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2fc9f-5f789b6cdc-sdwdq" podStartSLOduration=2.271739773 podStartE2EDuration="2.271739773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:38.270463777 +0000 UTC m=+1615.965819250" watchObservedRunningTime="2026-11-02 17:56:38.271739773 +0000 UTC m=+1615.967095237" Nov 02 17:56:39.187243 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:56:39.187201 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:56:49.193506 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:49.193472 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b2fc9f-6598c4cdc5-f5rrt" Nov 02 17:56:49.194038 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:49.193542 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2fc9f-5f789b6cdc-sdwdq" Nov 02 17:56:53.106685 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:53.106646 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8"] Nov 02 17:56:53.114764 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:53.114728 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8"] Nov 02 17:56:53.114930 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:53.114819 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" Nov 02 17:56:53.141350 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:53.141297 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2f6787\"/\"kube-root-ca.crt\"" Nov 02 17:56:53.141463 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:53.141415 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2f6787\"/\"openshift-service-ca.crt\"" Nov 02 17:56:53.154154 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:53.154130 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2f6787\"/\"default-dockercfg-mfv7j\"" Nov 02 17:56:53.154280 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:53.154251 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2f6787\"/\"test-model-2f6787-kserve-self-signed-certs\"" Nov 02 17:56:53.218545 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:53.218508 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33601c55-3e8f-41d4-a13e-1099a26334f4-tls-certs\") pod \"test-model-2f6787-kserve-c77466787-vn7z8\" (UID: \"33601c55-3e8f-41d4-a13e-1099a26334f4\") " pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" Nov 02 17:56:53.218739 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:53.218657 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33601c55-3e8f-41d4-a13e-1099a26334f4-dshm\") pod \"test-model-2f6787-kserve-c77466787-vn7z8\" (UID: \"33601c55-3e8f-41d4-a13e-1099a26334f4\") " pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" Nov 02 17:56:53.218739 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:53.218682 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smndx\" (UniqueName: \"kubernetes.io/projected/33601c55-3e8f-41d4-a13e-1099a26334f4-kube-api-access-smndx\") pod \"test-model-2f6787-kserve-c77466787-vn7z8\" (UID: \"33601c55-3e8f-41d4-a13e-1099a26334f4\") " pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" Nov 02 17:56:53.218849 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:53.218801 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33601c55-3e8f-41d4-a13e-1099a26334f4-tmp-dir\") pod \"test-model-2f6787-kserve-c77466787-vn7z8\" (UID: \"33601c55-3e8f-41d4-a13e-1099a26334f4\") " pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" Nov 02 17:56:53.218948 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:53.218928 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33601c55-3e8f-41d4-a13e-1099a26334f4-model-cache\") pod \"test-model-2f6787-kserve-c77466787-vn7z8\" (UID: \"33601c55-3e8f-41d4-a13e-1099a26334f4\") " pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" Nov 02 17:56:53.219031 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:53.219015 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33601c55-3e8f-41d4-a13e-1099a26334f4-home\") pod \"test-model-2f6787-kserve-c77466787-vn7z8\" (UID: \"33601c55-3e8f-41d4-a13e-1099a26334f4\") " pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" Nov 02 17:56:53.320926 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:53.320886 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33601c55-3e8f-41d4-a13e-1099a26334f4-dshm\") pod \"test-model-2f6787-kserve-c77466787-vn7z8\" (UID: \"33601c55-3e8f-41d4-a13e-1099a26334f4\") " pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" Nov 02 17:56:53.320926 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:53.320940 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smndx\" (UniqueName: \"kubernetes.io/projected/33601c55-3e8f-41d4-a13e-1099a26334f4-kube-api-access-smndx\") pod \"test-model-2f6787-kserve-c77466787-vn7z8\" (UID: \"33601c55-3e8f-41d4-a13e-1099a26334f4\") " pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" Nov 02 17:56:53.321159 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:53.321008 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33601c55-3e8f-41d4-a13e-1099a26334f4-tmp-dir\") pod \"test-model-2f6787-kserve-c77466787-vn7z8\" (UID: \"33601c55-3e8f-41d4-a13e-1099a26334f4\") " pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" Nov 02 17:56:53.321196 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:53.321159 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33601c55-3e8f-41d4-a13e-1099a26334f4-model-cache\") pod \"test-model-2f6787-kserve-c77466787-vn7z8\" (UID: \"33601c55-3e8f-41d4-a13e-1099a26334f4\") " pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" Nov 02 17:56:53.321277 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:53.321259 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33601c55-3e8f-41d4-a13e-1099a26334f4-home\") pod \"test-model-2f6787-kserve-c77466787-vn7z8\" (UID: \"33601c55-3e8f-41d4-a13e-1099a26334f4\") " pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" Nov 02 17:56:53.321346 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:53.321328 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33601c55-3e8f-41d4-a13e-1099a26334f4-tls-certs\") pod \"test-model-2f6787-kserve-c77466787-vn7z8\" (UID: \"33601c55-3e8f-41d4-a13e-1099a26334f4\") " pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" Nov 02 17:56:53.321507 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:53.321484 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33601c55-3e8f-41d4-a13e-1099a26334f4-tmp-dir\") pod \"test-model-2f6787-kserve-c77466787-vn7z8\" (UID: \"33601c55-3e8f-41d4-a13e-1099a26334f4\") " pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" Nov 02 17:56:53.321702 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:53.321675 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33601c55-3e8f-41d4-a13e-1099a26334f4-model-cache\") pod \"test-model-2f6787-kserve-c77466787-vn7z8\" (UID: \"33601c55-3e8f-41d4-a13e-1099a26334f4\") " pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" Nov 02 17:56:53.321975 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:53.321952 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33601c55-3e8f-41d4-a13e-1099a26334f4-home\") pod \"test-model-2f6787-kserve-c77466787-vn7z8\" (UID: \"33601c55-3e8f-41d4-a13e-1099a26334f4\") " pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" Nov 02 17:56:53.326324 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:53.323646 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33601c55-3e8f-41d4-a13e-1099a26334f4-dshm\") pod \"test-model-2f6787-kserve-c77466787-vn7z8\" (UID: \"33601c55-3e8f-41d4-a13e-1099a26334f4\") " pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" Nov 02 17:56:53.326324 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:53.324154 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33601c55-3e8f-41d4-a13e-1099a26334f4-tls-certs\") pod \"test-model-2f6787-kserve-c77466787-vn7z8\" (UID: \"33601c55-3e8f-41d4-a13e-1099a26334f4\") " pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" Nov 02 17:56:53.376383 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:53.373113 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smndx\" (UniqueName: \"kubernetes.io/projected/33601c55-3e8f-41d4-a13e-1099a26334f4-kube-api-access-smndx\") pod \"test-model-2f6787-kserve-c77466787-vn7z8\" (UID: \"33601c55-3e8f-41d4-a13e-1099a26334f4\") " pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" Nov 02 17:56:53.432342 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:53.430119 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" Nov 02 17:56:53.951126 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:56:53.950586 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33601c55_3e8f_41d4_a13e_1099a26334f4.slice/crio-41a9757ced0af9ff2af61f779dce9ef06abfe2e7baa631b46c51933b9aee3941 WatchSource:0}: Error finding container 41a9757ced0af9ff2af61f779dce9ef06abfe2e7baa631b46c51933b9aee3941: Status 404 returned error can't find the container with id 41a9757ced0af9ff2af61f779dce9ef06abfe2e7baa631b46c51933b9aee3941 Nov 02 17:56:53.970976 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:53.969994 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8"] Nov 02 17:56:54.405505 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:54.405466 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" event={"ID":"33601c55-3e8f-41d4-a13e-1099a26334f4","Type":"ContainerStarted","Data":"b3bbb91b545645cd58ce56e89a345a6ccf85034c9e14cb1a55b31dcc2b7f21dc"} Nov 02 17:56:54.406090 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:54.405517 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" event={"ID":"33601c55-3e8f-41d4-a13e-1099a26334f4","Type":"ContainerStarted","Data":"41a9757ced0af9ff2af61f779dce9ef06abfe2e7baa631b46c51933b9aee3941"} Nov 02 17:56:54.431069 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:54.431031 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" Nov 02 17:56:54.453942 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:54.453888 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" podStartSLOduration=1.45387275 podStartE2EDuration="1.45387275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:54.449263663 +0000 UTC m=+1632.144619155" watchObservedRunningTime="2026-11-02 17:56:54.45387275 +0000 UTC m=+1632.149228268" Nov 02 17:56:58.448707 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:58.448602 4471 generic.go:356] "Generic (PLEG): container finished" podID="5a819a2f-01ae-45ba-8c60-f3382617c58c" containerID="ded9c25461a48a974f0c690ed9509607e191a2124ad1cd4bec1c3bea33eab9aa" exitCode=0 Nov 02 17:56:58.448707 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:58.448668 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" event={"ID":"5a819a2f-01ae-45ba-8c60-f3382617c58c","Type":"ContainerDied","Data":"ded9c25461a48a974f0c690ed9509607e191a2124ad1cd4bec1c3bea33eab9aa"} Nov 02 17:56:58.449325 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:58.449155 4471 scope.go:122] "RemoveContainer" containerID="ded9c25461a48a974f0c690ed9509607e191a2124ad1cd4bec1c3bea33eab9aa" Nov 02 17:56:58.451257 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:58.451225 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:56:58.451370 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:58.451331 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:56:58.451413 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:58.451376 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:56:59.497329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:59.493893 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" event={"ID":"5a819a2f-01ae-45ba-8c60-f3382617c58c","Type":"ContainerStarted","Data":"d996ec065a24dfcd958bc124245dbf1713a865f0934908b38dbb7be195e0bbbc"} Nov 02 17:56:59.853933 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:59.853899 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:56:59.877326 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:59.855264 4471 patch_prober.go:29] interesting pod/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 17:56:59.877326 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:56:59.855340 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" podUID="5a819a2f-01ae-45ba-8c60-f3382617c58c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 17:57:00.854318 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:00.854281 4471 patch_prober.go:29] interesting pod/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 17:57:00.854729 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:00.854365 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" podUID="5a819a2f-01ae-45ba-8c60-f3382617c58c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 17:57:01.854074 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:01.854039 4471 patch_prober.go:29] interesting pod/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 17:57:01.854254 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:01.854113 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" podUID="5a819a2f-01ae-45ba-8c60-f3382617c58c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 17:57:02.860343 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:02.858851 4471 patch_prober.go:29] interesting pod/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 17:57:02.860343 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:02.858946 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" podUID="5a819a2f-01ae-45ba-8c60-f3382617c58c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 17:57:03.016963 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:03.016926 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:57:03.431215 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:03.431180 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" Nov 02 17:57:03.443618 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:03.443593 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" Nov 02 17:57:03.619134 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:03.619108 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" Nov 02 17:57:03.854924 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:03.854885 4471 patch_prober.go:29] interesting pod/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 17:57:03.855233 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:03.855197 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" podUID="5a819a2f-01ae-45ba-8c60-f3382617c58c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 17:57:04.854706 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:04.854505 4471 patch_prober.go:29] interesting pod/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 17:57:04.854706 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:04.854565 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" podUID="5a819a2f-01ae-45ba-8c60-f3382617c58c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 17:57:05.868338 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:05.868287 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:57:05.872670 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:05.872637 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 17:57:08.163234 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:08.163198 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d5b905-6bc9bc867-zg4xp"] Nov 02 17:57:08.166993 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:08.166973 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5b905-6bc9bc867-zg4xp" Nov 02 17:57:08.188819 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:08.188797 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d5b905-6bc9bc867-zg4xp"] Nov 02 17:57:08.196176 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:08.196151 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d5b905\"" Nov 02 17:57:08.196387 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:08.196219 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d5b905-dockercfg-sfklt\"" Nov 02 17:57:08.304327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:08.285732 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmd42\" (UniqueName: \"kubernetes.io/projected/cbecd70b-00b8-40ac-9d1c-87043a9cc31d-kube-api-access-kmd42\") pod \"payload-pre-processing-e2e-inf-b-d5b905-6bc9bc867-zg4xp\" (UID: \"cbecd70b-00b8-40ac-9d1c-87043a9cc31d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5b905-6bc9bc867-zg4xp" Nov 02 17:57:08.319327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:08.304732 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbecd70b-00b8-40ac-9d1c-87043a9cc31d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d5b905-6bc9bc867-zg4xp\" (UID: \"cbecd70b-00b8-40ac-9d1c-87043a9cc31d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5b905-6bc9bc867-zg4xp" Nov 02 17:57:08.438905 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:08.426397 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbecd70b-00b8-40ac-9d1c-87043a9cc31d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d5b905-6bc9bc867-zg4xp\" (UID: \"cbecd70b-00b8-40ac-9d1c-87043a9cc31d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5b905-6bc9bc867-zg4xp" Nov 02 17:57:08.438905 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:08.426786 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmd42\" (UniqueName: \"kubernetes.io/projected/cbecd70b-00b8-40ac-9d1c-87043a9cc31d-kube-api-access-kmd42\") pod \"payload-pre-processing-e2e-inf-b-d5b905-6bc9bc867-zg4xp\" (UID: \"cbecd70b-00b8-40ac-9d1c-87043a9cc31d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5b905-6bc9bc867-zg4xp" Nov 02 17:57:08.438905 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:08.428248 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbecd70b-00b8-40ac-9d1c-87043a9cc31d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d5b905-6bc9bc867-zg4xp\" (UID: \"cbecd70b-00b8-40ac-9d1c-87043a9cc31d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5b905-6bc9bc867-zg4xp" Nov 02 17:57:08.590108 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:08.590063 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmd42\" (UniqueName: \"kubernetes.io/projected/cbecd70b-00b8-40ac-9d1c-87043a9cc31d-kube-api-access-kmd42\") pod \"payload-pre-processing-e2e-inf-b-d5b905-6bc9bc867-zg4xp\" (UID: \"cbecd70b-00b8-40ac-9d1c-87043a9cc31d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5b905-6bc9bc867-zg4xp" Nov 02 17:57:08.781324 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:08.780711 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5b905-6bc9bc867-zg4xp" Nov 02 17:57:09.351365 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:09.351326 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d5b905-6bc9bc867-zg4xp"] Nov 02 17:57:09.483271 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:57:09.481728 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:57:09.693327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:09.693219 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5b905-6bc9bc867-zg4xp" event={"ID":"cbecd70b-00b8-40ac-9d1c-87043a9cc31d","Type":"ContainerStarted","Data":"e3ccd7b2023be19eca09a9a42f3d20b9f7dcba91913c87562613086febf52039"} Nov 02 17:57:09.693327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:09.693284 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5b905-6bc9bc867-zg4xp" event={"ID":"cbecd70b-00b8-40ac-9d1c-87043a9cc31d","Type":"ContainerStarted","Data":"61cb764fb4d6eb581cb748f399921f9f10d2a0595403568e9b1444a771676c99"} Nov 02 17:57:09.693562 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:09.693455 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5b905-6bc9bc867-zg4xp" Nov 02 17:57:09.776331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:09.773009 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5b905-6bc9bc867-zg4xp" podStartSLOduration=1.7729904090000002 podStartE2EDuration="1.772990409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:09.752861826 +0000 UTC m=+1647.448217327" watchObservedRunningTime="2026-11-02 17:57:09.772990409 +0000 UTC m=+1647.468345882" Nov 02 17:57:15.664332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:15.636152 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:57:20.709046 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:20.709014 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5b905-6bc9bc867-zg4xp" Nov 02 17:57:21.491676 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:21.491636 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn"] Nov 02 17:57:21.496464 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:21.496437 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" Nov 02 17:57:21.526893 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:21.526824 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8a209f\"/\"kube-root-ca.crt\"" Nov 02 17:57:21.526893 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:21.526824 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8a209f\"/\"default-dockercfg-hkdtg\"" Nov 02 17:57:21.527167 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:21.527105 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8a209f\"/\"openshift-service-ca.crt\"" Nov 02 17:57:21.527260 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:21.527204 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8a209f\"/\"test-model-8a209f-kserve-self-signed-certs\"" Nov 02 17:57:21.532457 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:21.532430 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn"] Nov 02 17:57:21.691111 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:21.691075 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-dshm\") pod \"test-model-8a209f-kserve-6544f7897f-r69cn\" (UID: \"b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2\") " pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" Nov 02 17:57:21.691409 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:21.691387 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-tmp-dir\") pod \"test-model-8a209f-kserve-6544f7897f-r69cn\" (UID: \"b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2\") " pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" Nov 02 17:57:21.691471 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:21.691440 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-tls-certs\") pod \"test-model-8a209f-kserve-6544f7897f-r69cn\" (UID: \"b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2\") " pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" Nov 02 17:57:21.691600 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:21.691584 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-model-cache\") pod \"test-model-8a209f-kserve-6544f7897f-r69cn\" (UID: \"b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2\") " pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" Nov 02 17:57:21.691729 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:21.691712 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp7m6\" (UniqueName: \"kubernetes.io/projected/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-kube-api-access-pp7m6\") pod \"test-model-8a209f-kserve-6544f7897f-r69cn\" (UID: \"b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2\") " pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" Nov 02 17:57:21.691794 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:21.691780 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-home\") pod \"test-model-8a209f-kserve-6544f7897f-r69cn\" (UID: \"b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2\") " pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" Nov 02 17:57:21.799404 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:21.799334 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-tmp-dir\") pod \"test-model-8a209f-kserve-6544f7897f-r69cn\" (UID: \"b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2\") " pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" Nov 02 17:57:21.799404 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:21.799402 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-tls-certs\") pod \"test-model-8a209f-kserve-6544f7897f-r69cn\" (UID: \"b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2\") " pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" Nov 02 17:57:21.799955 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:21.799490 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-model-cache\") pod \"test-model-8a209f-kserve-6544f7897f-r69cn\" (UID: \"b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2\") " pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" Nov 02 17:57:21.799955 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:21.799606 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp7m6\" (UniqueName: \"kubernetes.io/projected/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-kube-api-access-pp7m6\") pod \"test-model-8a209f-kserve-6544f7897f-r69cn\" (UID: \"b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2\") " pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" Nov 02 17:57:21.799955 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:21.799670 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-home\") pod \"test-model-8a209f-kserve-6544f7897f-r69cn\" (UID: \"b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2\") " pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" Nov 02 17:57:21.799955 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:21.799711 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-dshm\") pod \"test-model-8a209f-kserve-6544f7897f-r69cn\" (UID: \"b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2\") " pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" Nov 02 17:57:21.804332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:21.800569 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-model-cache\") pod \"test-model-8a209f-kserve-6544f7897f-r69cn\" (UID: \"b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2\") " pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" Nov 02 17:57:21.804332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:21.800857 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-tmp-dir\") pod \"test-model-8a209f-kserve-6544f7897f-r69cn\" (UID: \"b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2\") " pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" Nov 02 17:57:21.804930 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:21.804902 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-home\") pod \"test-model-8a209f-kserve-6544f7897f-r69cn\" (UID: \"b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2\") " pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" Nov 02 17:57:21.839986 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:21.839950 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-tls-certs\") pod \"test-model-8a209f-kserve-6544f7897f-r69cn\" (UID: \"b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2\") " pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" Nov 02 17:57:21.840216 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:21.840036 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp7m6\" (UniqueName: \"kubernetes.io/projected/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-kube-api-access-pp7m6\") pod \"test-model-8a209f-kserve-6544f7897f-r69cn\" (UID: \"b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2\") " pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" Nov 02 17:57:21.848330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:21.844642 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-dshm\") pod \"test-model-8a209f-kserve-6544f7897f-r69cn\" (UID: \"b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2\") " pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" Nov 02 17:57:22.124714 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:22.124625 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" Nov 02 17:57:22.485406 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:22.485364 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn"] Nov 02 17:57:22.485705 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:57:22.485682 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6aeceaa_bdb6_4f63_a0ea_f18cbc55bee2.slice/crio-154437eeab170def1fbf3a73d6df720f12164b5938639ad4430b2a78b8a95aa6 WatchSource:0}: Error finding container 154437eeab170def1fbf3a73d6df720f12164b5938639ad4430b2a78b8a95aa6: Status 404 returned error can't find the container with id 154437eeab170def1fbf3a73d6df720f12164b5938639ad4430b2a78b8a95aa6 Nov 02 17:57:23.012502 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:23.012456 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" event={"ID":"b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2","Type":"ContainerStarted","Data":"df8f007263a19f356a22aae99a7e61eaa8ff775b92a167473a2e57f1add8ef65"} Nov 02 17:57:23.012502 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:23.012506 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" event={"ID":"b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2","Type":"ContainerStarted","Data":"154437eeab170def1fbf3a73d6df720f12164b5938639ad4430b2a78b8a95aa6"} Nov 02 17:57:23.084798 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:23.084744 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" podStartSLOduration=2.084728159 podStartE2EDuration="2.084728159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:23.080729944 +0000 UTC m=+1660.776085418" watchObservedRunningTime="2026-11-02 17:57:23.084728159 +0000 UTC m=+1660.780083663" Nov 02 17:57:23.126542 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:23.126509 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" Nov 02 17:57:28.743861 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:28.743825 4471 patch_prober.go:29] interesting pod/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 17:57:28.744273 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:28.743879 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" podUID="19632b35-0e41-4d90-89c3-ced789733fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 17:57:29.073831 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:29.073792 4471 generic.go:356] "Generic (PLEG): container finished" podID="19632b35-0e41-4d90-89c3-ced789733fbc" containerID="f9796c87e49aaa1a9c983d5d0155d4d5208bc74f1f4deef29a67a86286af22b7" exitCode=0 Nov 02 17:57:29.074008 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:29.073874 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" event={"ID":"19632b35-0e41-4d90-89c3-ced789733fbc","Type":"ContainerDied","Data":"f9796c87e49aaa1a9c983d5d0155d4d5208bc74f1f4deef29a67a86286af22b7"} Nov 02 17:57:29.074542 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:29.074519 4471 scope.go:122] "RemoveContainer" containerID="f9796c87e49aaa1a9c983d5d0155d4d5208bc74f1f4deef29a67a86286af22b7" Nov 02 17:57:29.076707 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:29.076676 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:57:29.076820 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:29.076762 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:57:29.076820 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:29.076814 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:57:30.088589 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:30.088551 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" event={"ID":"19632b35-0e41-4d90-89c3-ced789733fbc","Type":"ContainerStarted","Data":"f2aae250a184d3f9120231114c61a5f8aceebfd1e61d53603be12066a16e1b01"} Nov 02 17:57:30.985790 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:30.977709 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:57:30.985790 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:30.977841 4471 patch_prober.go:29] interesting pod/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 17:57:30.985790 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:30.977886 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" podUID="19632b35-0e41-4d90-89c3-ced789733fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 17:57:31.962331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:31.961609 4471 patch_prober.go:29] interesting pod/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 17:57:31.962331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:31.961680 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" podUID="19632b35-0e41-4d90-89c3-ced789733fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 17:57:32.126055 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:32.126023 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" Nov 02 17:57:32.139131 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:32.139101 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" Nov 02 17:57:32.974343 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:32.970679 4471 patch_prober.go:29] interesting pod/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 17:57:32.974343 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:32.970758 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" podUID="19632b35-0e41-4d90-89c3-ced789733fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 17:57:33.122382 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:33.122352 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" Nov 02 17:57:33.967337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:33.964630 4471 patch_prober.go:29] interesting pod/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 17:57:33.967337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:33.964703 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" podUID="19632b35-0e41-4d90-89c3-ced789733fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 17:57:34.969330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:34.964667 4471 patch_prober.go:29] interesting pod/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 17:57:34.969330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:34.964730 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" podUID="19632b35-0e41-4d90-89c3-ced789733fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 17:57:35.963842 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:35.963809 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:57:35.964148 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:35.964125 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:57:35.964693 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:35.964675 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 17:57:39.184996 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:57:39.184953 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:57:59.996658 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:57:59.996617 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj"] Nov 02 17:58:00.001193 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:00.001169 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj" Nov 02 17:58:00.032092 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:00.032055 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-2c9b1d-serving-cert\"" Nov 02 17:58:00.037110 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:00.037075 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj"] Nov 02 17:58:00.065016 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:00.064976 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a186e1c4-6552-487a-bb08-3bd479c1b1bf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj\" (UID: \"a186e1c4-6552-487a-bb08-3bd479c1b1bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj" Nov 02 17:58:00.065222 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:00.065156 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a186e1c4-6552-487a-bb08-3bd479c1b1bf-metrics-certs\") pod \"maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj\" (UID: \"a186e1c4-6552-487a-bb08-3bd479c1b1bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj" Nov 02 17:58:00.065222 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:00.065192 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5ggp\" (UniqueName: \"kubernetes.io/projected/a186e1c4-6552-487a-bb08-3bd479c1b1bf-kube-api-access-r5ggp\") pod \"maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj\" (UID: \"a186e1c4-6552-487a-bb08-3bd479c1b1bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj" Nov 02 17:58:00.065395 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:00.065355 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a186e1c4-6552-487a-bb08-3bd479c1b1bf-maas-api-tls\") pod \"maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj\" (UID: \"a186e1c4-6552-487a-bb08-3bd479c1b1bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj" Nov 02 17:58:00.167447 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:00.167409 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a186e1c4-6552-487a-bb08-3bd479c1b1bf-metrics-certs\") pod \"maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj\" (UID: \"a186e1c4-6552-487a-bb08-3bd479c1b1bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj" Nov 02 17:58:00.167447 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:00.167471 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5ggp\" (UniqueName: \"kubernetes.io/projected/a186e1c4-6552-487a-bb08-3bd479c1b1bf-kube-api-access-r5ggp\") pod \"maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj\" (UID: \"a186e1c4-6552-487a-bb08-3bd479c1b1bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj" Nov 02 17:58:00.167772 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:00.167535 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a186e1c4-6552-487a-bb08-3bd479c1b1bf-maas-api-tls\") pod \"maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj\" (UID: \"a186e1c4-6552-487a-bb08-3bd479c1b1bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj" Nov 02 17:58:00.167772 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:58:00.167683 4471 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-2c9b1d-serving-cert: secret "maas-api-e2e-inf-b-2c9b1d-serving-cert" not found Nov 02 17:58:00.167885 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:58:00.167786 4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a186e1c4-6552-487a-bb08-3bd479c1b1bf-maas-api-tls podName:a186e1c4-6552-487a-bb08-3bd479c1b1bf nodeName:}" failed. No retries permitted until 2026-11-02 17:58:00.667753713 +0000 UTC m=+1698.363109188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a186e1c4-6552-487a-bb08-3bd479c1b1bf-maas-api-tls") pod "maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj" (UID: "a186e1c4-6552-487a-bb08-3bd479c1b1bf") : secret "maas-api-e2e-inf-b-2c9b1d-serving-cert" not found Nov 02 17:58:00.167954 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:00.167924 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a186e1c4-6552-487a-bb08-3bd479c1b1bf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj\" (UID: \"a186e1c4-6552-487a-bb08-3bd479c1b1bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj" Nov 02 17:58:00.168767 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:00.168741 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a186e1c4-6552-487a-bb08-3bd479c1b1bf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj\" (UID: \"a186e1c4-6552-487a-bb08-3bd479c1b1bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj" Nov 02 17:58:00.170256 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:00.170236 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a186e1c4-6552-487a-bb08-3bd479c1b1bf-metrics-certs\") pod \"maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj\" (UID: \"a186e1c4-6552-487a-bb08-3bd479c1b1bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj" Nov 02 17:58:00.239185 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:00.239103 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5ggp\" (UniqueName: \"kubernetes.io/projected/a186e1c4-6552-487a-bb08-3bd479c1b1bf-kube-api-access-r5ggp\") pod \"maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj\" (UID: \"a186e1c4-6552-487a-bb08-3bd479c1b1bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj" Nov 02 17:58:00.677770 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:00.677737 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a186e1c4-6552-487a-bb08-3bd479c1b1bf-maas-api-tls\") pod \"maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj\" (UID: \"a186e1c4-6552-487a-bb08-3bd479c1b1bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj" Nov 02 17:58:00.680280 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:00.680256 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a186e1c4-6552-487a-bb08-3bd479c1b1bf-maas-api-tls\") pod \"maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj\" (UID: \"a186e1c4-6552-487a-bb08-3bd479c1b1bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj" Nov 02 17:58:00.915146 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:00.915100 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj" Nov 02 17:58:01.408357 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:58:01.406614 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda186e1c4_6552_487a_bb08_3bd479c1b1bf.slice/crio-d3133b1de006acac30651ec49b1a0e910b7cb13361db8dbf443c65dc310da001 WatchSource:0}: Error finding container d3133b1de006acac30651ec49b1a0e910b7cb13361db8dbf443c65dc310da001: Status 404 returned error can't find the container with id d3133b1de006acac30651ec49b1a0e910b7cb13361db8dbf443c65dc310da001 Nov 02 17:58:01.408357 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:01.408226 4471 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:58:01.410189 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:01.410163 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj"] Nov 02 17:58:01.480071 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:01.480038 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj" event={"ID":"a186e1c4-6552-487a-bb08-3bd479c1b1bf","Type":"ContainerStarted","Data":"d3133b1de006acac30651ec49b1a0e910b7cb13361db8dbf443c65dc310da001"} Nov 02 17:58:02.489670 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:02.489625 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj" event={"ID":"a186e1c4-6552-487a-bb08-3bd479c1b1bf","Type":"ContainerStarted","Data":"6acb8d9f617be5f7080e9de7d6e7dde2ec18aeb1ce196561aec72d6e4f3e3502"} Nov 02 17:58:02.490205 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:02.489694 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj" Nov 02 17:58:02.538492 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:02.538439 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj" podStartSLOduration=3.126155422 podStartE2EDuration="3.538425298s" totalImagesPullingTime="412.269876ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:58:02.533606591 +0000 UTC m=+1700.228962065" watchObservedRunningTime="2026-11-02 17:58:02.538425298 +0000 UTC m=+1700.233780802" Nov 02 17:58:06.883331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:06.878457 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:58:06.883331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:06.879449 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 02 17:58:06.969973 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:06.969831 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:58:07.035260 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:07.035203 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:58:07.098460 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:07.098400 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" podUID="198ca016-6343-4dda-999d-63a39d349f3a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:58:07.205499 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:07.205394 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:58:07.347840 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:07.346737 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:58:07.375426 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:07.375380 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:58:07.491385 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:07.487369 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded" Nov 02 17:58:07.528447 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:07.528396 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:58:07.623500 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:07.622916 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" podUID="33601c55-3e8f-41d4-a13e-1099a26334f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.151:8000/ready\": context deadline exceeded" Nov 02 17:58:07.883520 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:07.883472 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:58:07.884595 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:07.884517 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 02 17:58:07.885405 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:07.885385 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:58:07.887667 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:07.887651 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:58:07.974332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:07.973393 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:58:07.975769 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:07.974821 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:58:08.047635 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:08.047593 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:58:08.048433 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:08.048412 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:58:08.210386 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:08.135643 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" podUID="b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.153:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:58:08.210386 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:08.136721 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" podUID="198ca016-6343-4dda-999d-63a39d349f3a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:58:08.210386 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:08.146043 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" Nov 02 17:58:08.210386 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:08.207409 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:58:08.210386 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:08.207980 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:58:08.331464 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:08.274209 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 17:58:08.413799 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:08.413758 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:58:08.414459 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:08.414436 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:58:08.421539 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:08.421327 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded" Nov 02 17:58:08.448554 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:08.448518 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:58:08.503207 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:08.503107 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 17:58:08.503207 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:08.503178 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 17:58:08.503546 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:08.503221 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 17:58:08.503546 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:08.503256 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" Nov 02 17:58:08.508992 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:08.508962 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj" Nov 02 17:58:08.515860 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:08.515824 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 17:58:08.518251 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:08.518161 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 17:58:09.206718 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:58:09.206672 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:58:15.814567 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:15.814534 4471 generic.go:356] "Generic (PLEG): container finished" podID="2c047abd-2c48-4b50-ba99-579d0f2cc3fd" containerID="15d32558f6a2658c05e1570f0d584c3a07008d0a460897a472f0f80216cfe32b" exitCode=0 Nov 02 17:58:15.815002 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:15.814596 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" event={"ID":"2c047abd-2c48-4b50-ba99-579d0f2cc3fd","Type":"ContainerDied","Data":"15d32558f6a2658c05e1570f0d584c3a07008d0a460897a472f0f80216cfe32b"} Nov 02 17:58:15.815081 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:15.815066 4471 scope.go:122] "RemoveContainer" containerID="15d32558f6a2658c05e1570f0d584c3a07008d0a460897a472f0f80216cfe32b" Nov 02 17:58:15.817091 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:15.817059 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:58:15.817189 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:15.817129 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:58:15.817189 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:15.817160 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:58:16.824137 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:16.824101 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" event={"ID":"2c047abd-2c48-4b50-ba99-579d0f2cc3fd","Type":"ContainerStarted","Data":"d5458af601b3e44d264665e61a48bcf0f5f2493bbb40b4dbac1f9bd29c16d58e"} Nov 02 17:58:17.671329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:17.670586 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:58:17.674375 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:17.674343 4471 patch_prober.go:29] interesting pod/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 17:58:17.674632 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:17.674602 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" podUID="2c047abd-2c48-4b50-ba99-579d0f2cc3fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 17:58:18.130125 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:18.130093 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:58:18.680337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:18.679365 4471 patch_prober.go:29] interesting pod/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 17:58:18.680337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:18.679439 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" podUID="2c047abd-2c48-4b50-ba99-579d0f2cc3fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 17:58:19.674715 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:19.674673 4471 patch_prober.go:29] interesting pod/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 17:58:19.675347 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:19.675293 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" podUID="2c047abd-2c48-4b50-ba99-579d0f2cc3fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 17:58:20.674330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:20.671693 4471 patch_prober.go:29] interesting pod/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 17:58:20.674330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:20.671770 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" podUID="2c047abd-2c48-4b50-ba99-579d0f2cc3fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 17:58:21.673336 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:21.668686 4471 patch_prober.go:29] interesting pod/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 17:58:21.673336 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:21.668753 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" podUID="2c047abd-2c48-4b50-ba99-579d0f2cc3fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 17:58:22.673564 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:22.673528 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:58:22.675990 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:22.675963 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 17:58:39.189073 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:58:39.189030 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:58:48.915587 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:48.915546 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r"] Nov 02 17:58:48.916149 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:48.915827 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" podUID="c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd" containerName="istio-proxy" containerID="cri-o://2d7f54c0b0acf24b5edbaa375a560d5d2cc71593c98d66c99eccca24b7b9dbdc" gracePeriod=30 Nov 02 17:58:51.878017 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:51.877980 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5"] Nov 02 17:58:51.883374 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:51.883347 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" Nov 02 17:58:51.904510 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:51.904482 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5"] Nov 02 17:58:51.912357 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:51.912327 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bb6d2e\"/\"default-dockercfg-wkgmt\"" Nov 02 17:58:51.913022 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:51.912999 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bb6d2e\"/\"test-model-bb6d2e-kserve-self-signed-certs\"" Nov 02 17:58:51.978270 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:51.978234 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-tls-certs\") pod \"test-model-bb6d2e-kserve-6d74447d6f-9wrh5\" (UID: \"c3e6eb3c-0496-44c8-a45f-9b4c662756b1\") " pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" Nov 02 17:58:51.978270 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:51.978273 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqflw\" (UniqueName: \"kubernetes.io/projected/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-kube-api-access-bqflw\") pod \"test-model-bb6d2e-kserve-6d74447d6f-9wrh5\" (UID: \"c3e6eb3c-0496-44c8-a45f-9b4c662756b1\") " pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" Nov 02 17:58:51.978582 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:51.978560 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-dshm\") pod \"test-model-bb6d2e-kserve-6d74447d6f-9wrh5\" (UID: \"c3e6eb3c-0496-44c8-a45f-9b4c662756b1\") " pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" Nov 02 17:58:51.978637 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:51.978618 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-model-cache\") pod \"test-model-bb6d2e-kserve-6d74447d6f-9wrh5\" (UID: \"c3e6eb3c-0496-44c8-a45f-9b4c662756b1\") " pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" Nov 02 17:58:51.978699 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:51.978686 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-tmp-dir\") pod \"test-model-bb6d2e-kserve-6d74447d6f-9wrh5\" (UID: \"c3e6eb3c-0496-44c8-a45f-9b4c662756b1\") " pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" Nov 02 17:58:51.978772 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:51.978757 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-home\") pod \"test-model-bb6d2e-kserve-6d74447d6f-9wrh5\" (UID: \"c3e6eb3c-0496-44c8-a45f-9b4c662756b1\") " pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" Nov 02 17:58:51.981817 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:51.981794 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bb6d2e\"/\"openshift-service-ca.crt\"" Nov 02 17:58:51.981942 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:51.981818 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bb6d2e\"/\"kube-root-ca.crt\"" Nov 02 17:58:52.080033 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:52.079993 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-tmp-dir\") pod \"test-model-bb6d2e-kserve-6d74447d6f-9wrh5\" (UID: \"c3e6eb3c-0496-44c8-a45f-9b4c662756b1\") " pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" Nov 02 17:58:52.080217 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:52.080079 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-home\") pod \"test-model-bb6d2e-kserve-6d74447d6f-9wrh5\" (UID: \"c3e6eb3c-0496-44c8-a45f-9b4c662756b1\") " pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" Nov 02 17:58:52.080217 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:52.080115 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-tls-certs\") pod \"test-model-bb6d2e-kserve-6d74447d6f-9wrh5\" (UID: \"c3e6eb3c-0496-44c8-a45f-9b4c662756b1\") " pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" Nov 02 17:58:52.080217 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:52.080138 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqflw\" (UniqueName: \"kubernetes.io/projected/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-kube-api-access-bqflw\") pod \"test-model-bb6d2e-kserve-6d74447d6f-9wrh5\" (UID: \"c3e6eb3c-0496-44c8-a45f-9b4c662756b1\") " pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" Nov 02 17:58:52.080452 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:52.080434 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-dshm\") pod \"test-model-bb6d2e-kserve-6d74447d6f-9wrh5\" (UID: \"c3e6eb3c-0496-44c8-a45f-9b4c662756b1\") " pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" Nov 02 17:58:52.080531 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:52.080508 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-tmp-dir\") pod \"test-model-bb6d2e-kserve-6d74447d6f-9wrh5\" (UID: \"c3e6eb3c-0496-44c8-a45f-9b4c662756b1\") " pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" Nov 02 17:58:52.080596 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:52.080538 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-home\") pod \"test-model-bb6d2e-kserve-6d74447d6f-9wrh5\" (UID: \"c3e6eb3c-0496-44c8-a45f-9b4c662756b1\") " pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" Nov 02 17:58:52.080596 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:52.080512 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-model-cache\") pod \"test-model-bb6d2e-kserve-6d74447d6f-9wrh5\" (UID: \"c3e6eb3c-0496-44c8-a45f-9b4c662756b1\") " pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" Nov 02 17:58:52.080751 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:52.080735 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-model-cache\") pod \"test-model-bb6d2e-kserve-6d74447d6f-9wrh5\" (UID: \"c3e6eb3c-0496-44c8-a45f-9b4c662756b1\") " pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" Nov 02 17:58:52.082571 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:52.082540 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-dshm\") pod \"test-model-bb6d2e-kserve-6d74447d6f-9wrh5\" (UID: \"c3e6eb3c-0496-44c8-a45f-9b4c662756b1\") " pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" Nov 02 17:58:52.082747 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:52.082727 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-tls-certs\") pod \"test-model-bb6d2e-kserve-6d74447d6f-9wrh5\" (UID: \"c3e6eb3c-0496-44c8-a45f-9b4c662756b1\") " pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" Nov 02 17:58:52.104010 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:52.103985 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqflw\" (UniqueName: \"kubernetes.io/projected/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-kube-api-access-bqflw\") pod \"test-model-bb6d2e-kserve-6d74447d6f-9wrh5\" (UID: \"c3e6eb3c-0496-44c8-a45f-9b4c662756b1\") " pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" Nov 02 17:58:52.203057 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:52.202973 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" Nov 02 17:58:52.650730 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:52.650705 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5"] Nov 02 17:58:53.275766 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:53.275721 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" event={"ID":"c3e6eb3c-0496-44c8-a45f-9b4c662756b1","Type":"ContainerStarted","Data":"6007b74909a21a6e1b4eb5c94fd5a6539e8493b3dc0daef0a807dc9799c0a5e5"} Nov 02 17:58:53.276170 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:53.275782 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" event={"ID":"c3e6eb3c-0496-44c8-a45f-9b4c662756b1","Type":"ContainerStarted","Data":"cc31a17d9d429dc0632539ee49868c76d5820de016ea82b19ace4c1752b8ff69"} Nov 02 17:58:53.322377 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:53.322293 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" podStartSLOduration=2.322279754 podStartE2EDuration="2.322279754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:58:53.317903476 +0000 UTC m=+1751.013258970" watchObservedRunningTime="2026-11-02 17:58:53.322279754 +0000 UTC m=+1751.017635226" Nov 02 17:58:54.204337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.204289 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" Nov 02 17:58:54.212135 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.212109 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:58:54.286558 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.286520 4471 generic.go:356] "Generic (PLEG): container finished" podID="c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd" containerID="2d7f54c0b0acf24b5edbaa375a560d5d2cc71593c98d66c99eccca24b7b9dbdc" exitCode=0 Nov 02 17:58:54.287035 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.286575 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" Nov 02 17:58:54.287035 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.286603 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" event={"ID":"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd","Type":"ContainerDied","Data":"2d7f54c0b0acf24b5edbaa375a560d5d2cc71593c98d66c99eccca24b7b9dbdc"} Nov 02 17:58:54.287035 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.286683 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r" event={"ID":"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd","Type":"ContainerDied","Data":"c8415ffafdc2306e5913b2bc8c6ab33959e65dabf86930b78c7bd23c533b7f08"} Nov 02 17:58:54.287035 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.286717 4471 scope.go:122] "RemoveContainer" containerID="2d7f54c0b0acf24b5edbaa375a560d5d2cc71593c98d66c99eccca24b7b9dbdc" Nov 02 17:58:54.298843 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.298817 4471 scope.go:122] "RemoveContainer" containerID="2d7f54c0b0acf24b5edbaa375a560d5d2cc71593c98d66c99eccca24b7b9dbdc" Nov 02 17:58:54.299194 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:58:54.299168 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d7f54c0b0acf24b5edbaa375a560d5d2cc71593c98d66c99eccca24b7b9dbdc\": container with ID starting with 2d7f54c0b0acf24b5edbaa375a560d5d2cc71593c98d66c99eccca24b7b9dbdc not found: ID does not exist" containerID="2d7f54c0b0acf24b5edbaa375a560d5d2cc71593c98d66c99eccca24b7b9dbdc" Nov 02 17:58:54.299289 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.299205 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d7f54c0b0acf24b5edbaa375a560d5d2cc71593c98d66c99eccca24b7b9dbdc"} err="failed to get container status \"2d7f54c0b0acf24b5edbaa375a560d5d2cc71593c98d66c99eccca24b7b9dbdc\": rpc error: code = NotFound desc = could not find container \"2d7f54c0b0acf24b5edbaa375a560d5d2cc71593c98d66c99eccca24b7b9dbdc\": container with ID starting with 2d7f54c0b0acf24b5edbaa375a560d5d2cc71593c98d66c99eccca24b7b9dbdc not found: ID does not exist" Nov 02 17:58:54.350358 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.350294 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-kube-api-access-62smk\" (UniqueName: \"kubernetes.io/projected/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-kube-api-access-62smk\") pod \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " Nov 02 17:58:54.350523 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.350501 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istio-envoy\") pod \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " Nov 02 17:58:54.350588 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.350554 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istio-data\") pod \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " Nov 02 17:58:54.350667 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.350641 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istio-podinfo\") pod \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " Nov 02 17:58:54.350722 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.350687 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istio-token\" (UniqueName: \"kubernetes.io/projected/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istio-token\") pod \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " Nov 02 17:58:54.350798 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.350780 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-credential-socket\") pod \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " Nov 02 17:58:54.351648 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.350865 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istiod-ca-cert\") pod \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " Nov 02 17:58:54.351648 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.350924 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istio-data" pod "c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd" (UID: "c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:54.351648 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.351155 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-credential-socket" pod "c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd" (UID: "c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:54.351648 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.351167 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-workload-socket\") pod \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " Nov 02 17:58:54.351648 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.351202 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-workload-certs\") pod \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\" (UID: \"c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd\") " Nov 02 17:58:54.351648 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.351238 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istiod-ca-cert" pod "c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd" (UID: "c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:58:54.351648 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.351411 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-workload-socket" pod "c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd" (UID: "c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:54.351648 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.351529 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-workload-certs" pod "c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd" (UID: "c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:54.352158 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.352069 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:58:54.352158 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.352091 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:58:54.352158 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.352108 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:58:54.352158 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.352121 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:58:54.352158 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.352134 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:58:54.353454 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.353237 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istio-envoy" pod "c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd" (UID: "c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:54.353989 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.353960 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-kube-api-access-62smk" pod "c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd" (UID: "c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd"). InnerVolumeSpecName "kube-api-access-62smk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:54.354240 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.354220 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istio-podinfo" pod "c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd" (UID: "c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:58:54.354773 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.354750 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istio-token" pod "c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd" (UID: "c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:54.458395 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.455836 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:58:54.458395 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.455877 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:58:54.458395 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.455892 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:58:54.458395 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.455905 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62smk\" (UniqueName: \"kubernetes.io/projected/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd-kube-api-access-62smk\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:58:54.631536 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.631499 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 17:58:54.638425 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.638393 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r"] Nov 02 17:58:54.648782 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:54.648750 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-6cb9r"] Nov 02 17:58:56.478374 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:58:56.473248 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd" path="/var/lib/kubelet/pods/c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd/volumes" Nov 02 17:59:02.204254 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:02.204211 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" Nov 02 17:59:02.217003 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:02.216979 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" Nov 02 17:59:02.388380 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:02.388347 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" Nov 02 17:59:09.228846 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:59:09.228774 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:59:09.608609 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.608579 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2"] Nov 02 17:59:09.609439 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.609420 4471 cpu_manager.go:414] "containerMap: removing container" podUID="c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd" containerName="istio-proxy" Nov 02 17:59:09.609439 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.609439 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd" containerName="istio-proxy" Nov 02 17:59:09.609619 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.609609 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7a1e0e8-1ff4-4ba9-8b2e-fef4cb120afd" containerName="istio-proxy" Nov 02 17:59:09.612424 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.612406 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 17:59:09.629073 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.629042 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz"] Nov 02 17:59:09.632853 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.632833 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 17:59:09.635671 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.635648 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2c9b1d-openshift-default-dockercfg-v7bnw\"" Nov 02 17:59:09.645393 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.645370 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2"] Nov 02 17:59:09.661035 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.661004 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz"] Nov 02 17:59:09.723225 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.723185 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istio-data\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 17:59:09.723407 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.723319 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvnwl\" (UniqueName: \"kubernetes.io/projected/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-kube-api-access-jvnwl\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 17:59:09.723407 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.723364 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef139666-59eb-4847-9f3d-8fc0d20529dc-istio-token\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 17:59:09.723498 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.723408 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istio-token\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 17:59:09.723534 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.723497 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-workload-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 17:59:09.723588 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.723573 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-istio-envoy\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 17:59:09.723627 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.723609 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef139666-59eb-4847-9f3d-8fc0d20529dc-istiod-ca-cert\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 17:59:09.723661 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.723636 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istio-envoy\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 17:59:09.723758 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.723737 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-credential-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 17:59:09.723796 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.723774 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istio-podinfo\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 17:59:09.723882 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.723868 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-credential-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 17:59:09.723927 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.723894 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc9n4\" (UniqueName: \"kubernetes.io/projected/ef139666-59eb-4847-9f3d-8fc0d20529dc-kube-api-access-gc9n4\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 17:59:09.723962 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.723930 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef139666-59eb-4847-9f3d-8fc0d20529dc-istio-podinfo\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 17:59:09.723996 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.723983 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-istio-data\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 17:59:09.724030 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.724003 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istiod-ca-cert\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 17:59:09.724078 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.724063 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-workload-certs\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 17:59:09.724126 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.724101 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-workload-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 17:59:09.724172 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.724147 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-workload-certs\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 17:59:09.825737 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.825701 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-workload-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 17:59:09.825898 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.825750 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-workload-certs\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 17:59:09.825898 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.825829 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istio-data\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 17:59:09.825972 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.825924 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvnwl\" (UniqueName: \"kubernetes.io/projected/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-kube-api-access-jvnwl\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 17:59:09.826014 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.825979 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef139666-59eb-4847-9f3d-8fc0d20529dc-istio-token\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 17:59:09.826061 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.826044 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istio-token\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 17:59:09.826118 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.826097 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-workload-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 17:59:09.826206 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.826171 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-istio-envoy\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 17:59:09.826206 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.826173 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-workload-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 17:59:09.826206 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.826177 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-workload-certs\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 17:59:09.826386 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.826210 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef139666-59eb-4847-9f3d-8fc0d20529dc-istiod-ca-cert\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 17:59:09.826386 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.826222 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istio-data\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 17:59:09.826386 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.826254 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istio-envoy\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 17:59:09.826386 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.826346 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-credential-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 17:59:09.826386 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.826382 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istio-podinfo\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 17:59:09.826674 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.826491 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-workload-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 17:59:09.826674 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.826492 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-credential-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 17:59:09.826674 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.826543 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc9n4\" (UniqueName: \"kubernetes.io/projected/ef139666-59eb-4847-9f3d-8fc0d20529dc-kube-api-access-gc9n4\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 17:59:09.826674 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.826608 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef139666-59eb-4847-9f3d-8fc0d20529dc-istio-podinfo\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 17:59:09.826674 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.826643 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-istio-data\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 17:59:09.826674 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.826663 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-credential-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 17:59:09.827000 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.826675 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istiod-ca-cert\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 17:59:09.827000 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.826723 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-workload-certs\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 17:59:09.827000 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.826751 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-credential-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 17:59:09.827000 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.826915 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef139666-59eb-4847-9f3d-8fc0d20529dc-istiod-ca-cert\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 17:59:09.827223 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.827166 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-istio-data\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 17:59:09.827223 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.827215 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-workload-certs\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 17:59:09.827497 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.827475 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istiod-ca-cert\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 17:59:09.829045 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.829016 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-istio-envoy\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 17:59:09.829150 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.829019 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istio-envoy\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 17:59:09.829150 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.829029 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef139666-59eb-4847-9f3d-8fc0d20529dc-istio-podinfo\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 17:59:09.829223 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.829156 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istio-podinfo\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 17:59:09.850695 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.850668 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef139666-59eb-4847-9f3d-8fc0d20529dc-istio-token\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 17:59:09.850910 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.850888 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc9n4\" (UniqueName: \"kubernetes.io/projected/ef139666-59eb-4847-9f3d-8fc0d20529dc-kube-api-access-gc9n4\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 17:59:09.851134 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.851113 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istio-token\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 17:59:09.851238 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.851224 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvnwl\" (UniqueName: \"kubernetes.io/projected/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-kube-api-access-jvnwl\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 17:59:09.930738 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.930656 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 17:59:09.954625 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:09.954599 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 17:59:10.253486 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:59:10.253457 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8eeabf7_f670_46c2_bb59_e07e1a1f1337.slice/crio-6e554d9c682c3fb5296620d8dd51afa4a09b7d58eef694da70fd7b9de027b717 WatchSource:0}: Error finding container 6e554d9c682c3fb5296620d8dd51afa4a09b7d58eef694da70fd7b9de027b717: Status 404 returned error can't find the container with id 6e554d9c682c3fb5296620d8dd51afa4a09b7d58eef694da70fd7b9de027b717 Nov 02 17:59:10.255432 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:10.255406 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2"] Nov 02 17:59:10.256651 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:10.256618 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:59:10.256767 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:10.256695 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:59:10.256767 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:10.256741 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:59:10.279914 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:59:10.279890 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef139666_59eb_4847_9f3d_8fc0d20529dc.slice/crio-9c729e45cc9aa465b0d5dad972651eb2c93afe184d90bc9293c4526711295ade WatchSource:0}: Error finding container 9c729e45cc9aa465b0d5dad972651eb2c93afe184d90bc9293c4526711295ade: Status 404 returned error can't find the container with id 9c729e45cc9aa465b0d5dad972651eb2c93afe184d90bc9293c4526711295ade Nov 02 17:59:10.282193 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:10.282161 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz"] Nov 02 17:59:10.283004 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:10.282971 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:59:10.283084 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:10.283035 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:59:10.283084 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:10.283065 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:59:10.449296 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:10.449258 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" event={"ID":"e8eeabf7-f670-46c2-bb59-e07e1a1f1337","Type":"ContainerStarted","Data":"7cbcecb8519c03d1a8274e21d4df7ffdafc32accf5c8e9e3bf46e9114ecf7652"} Nov 02 17:59:10.449296 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:10.449354 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" event={"ID":"e8eeabf7-f670-46c2-bb59-e07e1a1f1337","Type":"ContainerStarted","Data":"6e554d9c682c3fb5296620d8dd51afa4a09b7d58eef694da70fd7b9de027b717"} Nov 02 17:59:10.451724 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:10.451689 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" event={"ID":"ef139666-59eb-4847-9f3d-8fc0d20529dc","Type":"ContainerStarted","Data":"63daf3d4ceb7310bf60553eb303ae9de8030019b0e38713356b0a95fe2bfc420"} Nov 02 17:59:10.451851 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:10.451745 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" event={"ID":"ef139666-59eb-4847-9f3d-8fc0d20529dc","Type":"ContainerStarted","Data":"9c729e45cc9aa465b0d5dad972651eb2c93afe184d90bc9293c4526711295ade"} Nov 02 17:59:10.502831 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:10.502723 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" podStartSLOduration=1.502707192 podStartE2EDuration="1.502707192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:10.498782926 +0000 UTC m=+1768.194138421" watchObservedRunningTime="2026-11-02 17:59:10.502707192 +0000 UTC m=+1768.198062664" Nov 02 17:59:10.550099 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:10.550034 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" podStartSLOduration=1.5500162 podStartE2EDuration="1.5500162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:10.545632467 +0000 UTC m=+1768.240987967" watchObservedRunningTime="2026-11-02 17:59:10.5500162 +0000 UTC m=+1768.245371674" Nov 02 17:59:10.934533 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:10.934446 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 17:59:10.955550 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:10.955471 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 17:59:11.933370 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:11.933329 4471 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 02 17:59:11.933808 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:11.933394 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" podUID="e8eeabf7-f670-46c2-bb59-e07e1a1f1337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 02 17:59:11.965337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:11.960883 4471 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 02 17:59:11.965337 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:11.960940 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" podUID="ef139666-59eb-4847-9f3d-8fc0d20529dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 02 17:59:12.931233 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:12.931199 4471 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 02 17:59:12.931502 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:12.931468 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" podUID="e8eeabf7-f670-46c2-bb59-e07e1a1f1337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 02 17:59:12.955665 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:12.955632 4471 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 02 17:59:12.956194 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:12.956155 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" podUID="ef139666-59eb-4847-9f3d-8fc0d20529dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 02 17:59:13.931637 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:13.931601 4471 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 02 17:59:13.931918 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:13.931889 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" podUID="e8eeabf7-f670-46c2-bb59-e07e1a1f1337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 02 17:59:13.955283 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:13.955255 4471 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 02 17:59:13.955471 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:13.955296 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" podUID="ef139666-59eb-4847-9f3d-8fc0d20529dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 02 17:59:14.931886 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:14.931849 4471 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 02 17:59:14.932294 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:14.931908 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" podUID="e8eeabf7-f670-46c2-bb59-e07e1a1f1337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 02 17:59:14.955190 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:14.955159 4471 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 02 17:59:14.955374 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:14.955211 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" podUID="ef139666-59eb-4847-9f3d-8fc0d20529dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 02 17:59:15.954329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:15.942762 4471 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 02 17:59:15.954329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:15.942861 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" podUID="e8eeabf7-f670-46c2-bb59-e07e1a1f1337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 02 17:59:15.962336 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:15.957798 4471 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 02 17:59:15.962336 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:15.957864 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" podUID="ef139666-59eb-4847-9f3d-8fc0d20529dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 02 17:59:16.937145 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:16.937114 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 17:59:16.937556 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:16.937533 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 17:59:16.938242 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:16.938223 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 17:59:16.960591 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:16.960568 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 17:59:16.960921 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:16.960835 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 17:59:16.961461 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:16.961443 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 17:59:17.900184 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:17.900139 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5"] Nov 02 17:59:17.904515 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:17.904492 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 17:59:17.935212 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:17.935182 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d5b905-openshift-default-dockercfg-vtgjv\"" Nov 02 17:59:17.944644 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:17.944619 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5"] Nov 02 17:59:17.979639 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:17.979609 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24f05804-b83e-44f0-b01d-b1f34904fba2-istiod-ca-cert\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 17:59:17.980065 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:17.979690 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-credential-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 17:59:17.980065 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:17.979816 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-workload-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 17:59:17.980065 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:17.979934 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb7qk\" (UniqueName: \"kubernetes.io/projected/24f05804-b83e-44f0-b01d-b1f34904fba2-kube-api-access-gb7qk\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 17:59:17.980065 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:17.980032 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-istio-data\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 17:59:17.980245 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:17.980156 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-istio-envoy\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 17:59:17.980245 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:17.980229 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-workload-certs\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 17:59:17.980404 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:17.980363 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24f05804-b83e-44f0-b01d-b1f34904fba2-istio-token\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 17:59:17.980532 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:17.980514 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24f05804-b83e-44f0-b01d-b1f34904fba2-istio-podinfo\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 17:59:18.082197 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:18.082156 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24f05804-b83e-44f0-b01d-b1f34904fba2-istio-podinfo\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 17:59:18.082415 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:18.082233 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24f05804-b83e-44f0-b01d-b1f34904fba2-istiod-ca-cert\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 17:59:18.082415 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:18.082296 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-credential-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 17:59:18.082415 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:18.082353 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-workload-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 17:59:18.082415 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:18.082400 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb7qk\" (UniqueName: \"kubernetes.io/projected/24f05804-b83e-44f0-b01d-b1f34904fba2-kube-api-access-gb7qk\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 17:59:18.082620 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:18.082473 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-istio-data\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 17:59:18.082620 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:18.082533 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-istio-envoy\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 17:59:18.082620 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:18.082606 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-workload-certs\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 17:59:18.082789 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:18.082646 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24f05804-b83e-44f0-b01d-b1f34904fba2-istio-token\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 17:59:18.083051 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:18.083020 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-credential-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 17:59:18.083238 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:18.083210 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-workload-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 17:59:18.083329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:18.083178 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24f05804-b83e-44f0-b01d-b1f34904fba2-istiod-ca-cert\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 17:59:18.083329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:18.083155 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-workload-certs\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 17:59:18.083329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:18.083275 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-istio-data\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 17:59:18.084934 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:18.084915 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-istio-envoy\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 17:59:18.085035 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:18.084931 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24f05804-b83e-44f0-b01d-b1f34904fba2-istio-podinfo\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 17:59:18.106639 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:18.106611 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24f05804-b83e-44f0-b01d-b1f34904fba2-istio-token\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 17:59:18.107526 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:18.107507 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb7qk\" (UniqueName: \"kubernetes.io/projected/24f05804-b83e-44f0-b01d-b1f34904fba2-kube-api-access-gb7qk\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 17:59:18.218541 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:18.218461 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 17:59:18.535515 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:59:18.535484 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f05804_b83e_44f0_b01d_b1f34904fba2.slice/crio-8ae25c8c6b88e8f87c6c6118a99caa7b1cd05343f08bfaa2cafaf2d28323c5b1 WatchSource:0}: Error finding container 8ae25c8c6b88e8f87c6c6118a99caa7b1cd05343f08bfaa2cafaf2d28323c5b1: Status 404 returned error can't find the container with id 8ae25c8c6b88e8f87c6c6118a99caa7b1cd05343f08bfaa2cafaf2d28323c5b1 Nov 02 17:59:18.537787 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:18.537762 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5"] Nov 02 17:59:18.538821 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:18.538788 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:59:18.538936 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:18.538873 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:59:18.538936 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:18.538925 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:59:18.641379 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:18.641297 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" event={"ID":"24f05804-b83e-44f0-b01d-b1f34904fba2","Type":"ContainerStarted","Data":"8786f64d678c887c83efbb605697260c7d8a7e437393df396e110efb1d948c58"} Nov 02 17:59:18.642110 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:18.642076 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" event={"ID":"24f05804-b83e-44f0-b01d-b1f34904fba2","Type":"ContainerStarted","Data":"8ae25c8c6b88e8f87c6c6118a99caa7b1cd05343f08bfaa2cafaf2d28323c5b1"} Nov 02 17:59:18.696848 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:18.696797 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" podStartSLOduration=1.69678236 podStartE2EDuration="1.69678236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:18.691099242 +0000 UTC m=+1776.386454714" watchObservedRunningTime="2026-11-02 17:59:18.69678236 +0000 UTC m=+1776.392137838" Nov 02 17:59:19.160803 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:19.160763 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd"] Nov 02 17:59:19.161273 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:19.161038 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" podUID="ec23f89c-765f-4c30-a3e5-761ca79f0288" containerName="istio-proxy" containerID="cri-o://bc1f92a2701a88d7785bca48ccbed6f62a183cda50a47468e285924dc4e2a0ae" gracePeriod=30 Nov 02 17:59:19.219060 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:19.218878 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 17:59:19.220366 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:19.220330 4471 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 02 17:59:19.220509 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:19.220421 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" podUID="24f05804-b83e-44f0-b01d-b1f34904fba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 02 17:59:20.219375 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:20.219339 4471 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 02 17:59:20.219913 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:20.219390 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" podUID="24f05804-b83e-44f0-b01d-b1f34904fba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 02 17:59:21.222344 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:21.219171 4471 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 02 17:59:21.222344 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:21.219244 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" podUID="24f05804-b83e-44f0-b01d-b1f34904fba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 02 17:59:22.189526 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.189474 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597"] Nov 02 17:59:22.198776 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.198729 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 17:59:22.215939 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.215908 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597"] Nov 02 17:59:22.218922 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.218889 4471 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 02 17:59:22.219144 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.218983 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" podUID="24f05804-b83e-44f0-b01d-b1f34904fba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 02 17:59:22.281658 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.281622 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-96768b-openshift-default-dockercfg-v8qql\"" Nov 02 17:59:22.291281 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.291245 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwp8r\" (UniqueName: \"kubernetes.io/projected/c1e9cee8-e915-4a1f-819d-226dcda7485a-kube-api-access-bwp8r\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-bt597\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 17:59:22.291281 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.291282 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e9cee8-e915-4a1f-819d-226dcda7485a-istio-podinfo\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-bt597\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 17:59:22.291532 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.291388 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-istio-data\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-bt597\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 17:59:22.291532 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.291417 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-workload-certs\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-bt597\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 17:59:22.291532 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.291493 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-credential-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-bt597\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 17:59:22.291706 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.291538 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e9cee8-e915-4a1f-819d-226dcda7485a-istiod-ca-cert\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-bt597\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 17:59:22.291706 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.291584 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-istio-envoy\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-bt597\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 17:59:22.291706 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.291624 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e9cee8-e915-4a1f-819d-226dcda7485a-istio-token\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-bt597\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 17:59:22.291706 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.291663 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-workload-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-bt597\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 17:59:22.392868 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.392833 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e9cee8-e915-4a1f-819d-226dcda7485a-istio-token\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-bt597\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 17:59:22.393060 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.392906 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-workload-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-bt597\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 17:59:22.393136 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.393086 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwp8r\" (UniqueName: \"kubernetes.io/projected/c1e9cee8-e915-4a1f-819d-226dcda7485a-kube-api-access-bwp8r\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-bt597\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 17:59:22.393136 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.393117 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e9cee8-e915-4a1f-819d-226dcda7485a-istio-podinfo\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-bt597\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 17:59:22.393237 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.393210 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-istio-data\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-bt597\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 17:59:22.393280 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.393248 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-workload-certs\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-bt597\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 17:59:22.393399 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.393377 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-workload-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-bt597\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 17:59:22.393443 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.393399 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-credential-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-bt597\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 17:59:22.393503 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.393488 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e9cee8-e915-4a1f-819d-226dcda7485a-istiod-ca-cert\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-bt597\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 17:59:22.393588 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.393564 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-istio-envoy\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-bt597\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 17:59:22.393667 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.393594 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-istio-data\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-bt597\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 17:59:22.393667 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.393621 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-workload-certs\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-bt597\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 17:59:22.393778 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.393692 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-credential-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-bt597\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 17:59:22.394096 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.394075 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e9cee8-e915-4a1f-819d-226dcda7485a-istiod-ca-cert\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-bt597\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 17:59:22.395666 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.395645 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-istio-envoy\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-bt597\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 17:59:22.395843 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.395819 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e9cee8-e915-4a1f-819d-226dcda7485a-istio-podinfo\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-bt597\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 17:59:22.416094 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.416055 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e9cee8-e915-4a1f-819d-226dcda7485a-istio-token\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-bt597\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 17:59:22.417135 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.417107 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwp8r\" (UniqueName: \"kubernetes.io/projected/c1e9cee8-e915-4a1f-819d-226dcda7485a-kube-api-access-bwp8r\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-bt597\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 17:59:22.516574 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.516545 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 17:59:22.883964 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.883928 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597"] Nov 02 17:59:22.884139 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:59:22.884094 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e9cee8_e915_4a1f_819d_226dcda7485a.slice/crio-42a6e06382c9794653f26594cb26be3397af216fe0c06df593264c8e28901888 WatchSource:0}: Error finding container 42a6e06382c9794653f26594cb26be3397af216fe0c06df593264c8e28901888: Status 404 returned error can't find the container with id 42a6e06382c9794653f26594cb26be3397af216fe0c06df593264c8e28901888 Nov 02 17:59:22.887596 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.887563 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:59:22.887701 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.887632 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:59:22.887701 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:22.887682 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:59:23.225765 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:23.225683 4471 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 02 17:59:23.225765 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:23.225744 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" podUID="24f05804-b83e-44f0-b01d-b1f34904fba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 02 17:59:23.795247 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:23.795197 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" event={"ID":"c1e9cee8-e915-4a1f-819d-226dcda7485a","Type":"ContainerStarted","Data":"18c0e4be2b5df625b39ee8e8cb2be8eb15ea519a0ee6d2ff9ff91e8f0c2dd0d8"} Nov 02 17:59:23.795908 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:23.795287 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" event={"ID":"c1e9cee8-e915-4a1f-819d-226dcda7485a","Type":"ContainerStarted","Data":"42a6e06382c9794653f26594cb26be3397af216fe0c06df593264c8e28901888"} Nov 02 17:59:23.853420 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:23.853356 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" podStartSLOduration=1.8533321520000001 podStartE2EDuration="1.853332152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:23.848005982 +0000 UTC m=+1781.543361479" watchObservedRunningTime="2026-11-02 17:59:23.853332152 +0000 UTC m=+1781.548687626" Nov 02 17:59:24.219063 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.219025 4471 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 02 17:59:24.219760 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.219283 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" podUID="24f05804-b83e-44f0-b01d-b1f34904fba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 02 17:59:24.516806 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.516781 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 17:59:24.518486 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.518376 4471 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 17:59:24.518486 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.518424 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" podUID="c1e9cee8-e915-4a1f-819d-226dcda7485a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 17:59:24.620951 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.620896 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 17:59:24.621141 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.620964 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" podUID="ec23f89c-765f-4c30-a3e5-761ca79f0288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 17:59:24.718332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.713461 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq"] Nov 02 17:59:24.740582 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.740552 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq"] Nov 02 17:59:24.740734 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.740713 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 17:59:24.826369 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.825330 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istio-podinfo\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 17:59:24.826369 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.825422 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-credential-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 17:59:24.826369 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.825621 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfxls\" (UniqueName: \"kubernetes.io/projected/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-kube-api-access-cfxls\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 17:59:24.826369 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.825677 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istio-data\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 17:59:24.826369 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.825771 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-workload-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 17:59:24.826369 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.825802 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istio-token\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 17:59:24.826369 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.825976 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-workload-certs\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 17:59:24.826369 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.826042 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istio-envoy\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 17:59:24.826369 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.826102 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istiod-ca-cert\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 17:59:24.831351 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.829985 4471 generic.go:356] "Generic (PLEG): container finished" podID="ec23f89c-765f-4c30-a3e5-761ca79f0288" containerID="bc1f92a2701a88d7785bca48ccbed6f62a183cda50a47468e285924dc4e2a0ae" exitCode=0 Nov 02 17:59:24.831351 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.831046 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" event={"ID":"ec23f89c-765f-4c30-a3e5-761ca79f0288","Type":"ContainerDied","Data":"bc1f92a2701a88d7785bca48ccbed6f62a183cda50a47468e285924dc4e2a0ae"} Nov 02 17:59:24.928896 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.928627 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfxls\" (UniqueName: \"kubernetes.io/projected/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-kube-api-access-cfxls\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 17:59:24.928896 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.928711 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istio-data\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 17:59:24.928896 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.928772 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-workload-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 17:59:24.928896 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.928802 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istio-token\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 17:59:24.929261 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.928927 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-workload-certs\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 17:59:24.929261 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.928996 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istio-envoy\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 17:59:24.929261 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.929067 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istiod-ca-cert\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 17:59:24.929261 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.929239 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istio-podinfo\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 17:59:24.929578 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.929400 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-credential-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 17:59:24.929889 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.929866 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istio-data\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 17:59:24.929975 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.929913 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-credential-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 17:59:24.930169 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.930143 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-workload-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 17:59:24.930632 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.930607 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istiod-ca-cert\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 17:59:24.930731 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.930694 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-workload-certs\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 17:59:24.933919 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.933891 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istio-envoy\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 17:59:24.934188 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.934158 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istio-podinfo\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 17:59:24.953114 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.953091 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfxls\" (UniqueName: \"kubernetes.io/projected/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-kube-api-access-cfxls\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 17:59:24.954398 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:24.954369 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istio-token\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 17:59:25.160872 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.160784 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 17:59:25.220796 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.218814 4471 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 02 17:59:25.220796 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.218863 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" podUID="24f05804-b83e-44f0-b01d-b1f34904fba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 02 17:59:25.240280 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.240011 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:59:25.338852 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.338439 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec23f89c-765f-4c30-a3e5-761ca79f0288-istio-token\" (UniqueName: \"kubernetes.io/projected/ec23f89c-765f-4c30-a3e5-761ca79f0288-istio-token\") pod \"ec23f89c-765f-4c30-a3e5-761ca79f0288\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " Nov 02 17:59:25.338852 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.338538 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-istio-data\") pod \"ec23f89c-765f-4c30-a3e5-761ca79f0288\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " Nov 02 17:59:25.338852 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.338592 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-workload-certs\") pod \"ec23f89c-765f-4c30-a3e5-761ca79f0288\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " Nov 02 17:59:25.338852 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.338687 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec23f89c-765f-4c30-a3e5-761ca79f0288-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec23f89c-765f-4c30-a3e5-761ca79f0288-istiod-ca-cert\") pod \"ec23f89c-765f-4c30-a3e5-761ca79f0288\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " Nov 02 17:59:25.338852 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.338755 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-istio-envoy\") pod \"ec23f89c-765f-4c30-a3e5-761ca79f0288\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " Nov 02 17:59:25.339280 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.338989 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec23f89c-765f-4c30-a3e5-761ca79f0288-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec23f89c-765f-4c30-a3e5-761ca79f0288-istio-podinfo\") pod \"ec23f89c-765f-4c30-a3e5-761ca79f0288\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " Nov 02 17:59:25.339280 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.339047 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-workload-certs" pod "ec23f89c-765f-4c30-a3e5-761ca79f0288" (UID: "ec23f89c-765f-4c30-a3e5-761ca79f0288"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:59:25.339280 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.339066 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec23f89c-765f-4c30-a3e5-761ca79f0288-kube-api-access-m95w2\" (UniqueName: \"kubernetes.io/projected/ec23f89c-765f-4c30-a3e5-761ca79f0288-kube-api-access-m95w2\") pod \"ec23f89c-765f-4c30-a3e5-761ca79f0288\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " Nov 02 17:59:25.339280 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.339173 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-credential-socket\") pod \"ec23f89c-765f-4c30-a3e5-761ca79f0288\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " Nov 02 17:59:25.339280 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.339222 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-istio-data" pod "ec23f89c-765f-4c30-a3e5-761ca79f0288" (UID: "ec23f89c-765f-4c30-a3e5-761ca79f0288"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:59:25.339280 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.339232 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-workload-socket\") pod \"ec23f89c-765f-4c30-a3e5-761ca79f0288\" (UID: \"ec23f89c-765f-4c30-a3e5-761ca79f0288\") " Nov 02 17:59:25.342365 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.340148 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:59:25.342365 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.340180 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:59:25.342365 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.341345 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec23f89c-765f-4c30-a3e5-761ca79f0288-istio-token" pod "ec23f89c-765f-4c30-a3e5-761ca79f0288" (UID: "ec23f89c-765f-4c30-a3e5-761ca79f0288"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:59:25.342365 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.341757 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-credential-socket" pod "ec23f89c-765f-4c30-a3e5-761ca79f0288" (UID: "ec23f89c-765f-4c30-a3e5-761ca79f0288"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:59:25.342365 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.342023 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec23f89c-765f-4c30-a3e5-761ca79f0288-istiod-ca-cert" pod "ec23f89c-765f-4c30-a3e5-761ca79f0288" (UID: "ec23f89c-765f-4c30-a3e5-761ca79f0288"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:59:25.342721 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.342399 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-workload-socket" pod "ec23f89c-765f-4c30-a3e5-761ca79f0288" (UID: "ec23f89c-765f-4c30-a3e5-761ca79f0288"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:59:25.354257 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.349535 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec23f89c-765f-4c30-a3e5-761ca79f0288-istio-podinfo" pod "ec23f89c-765f-4c30-a3e5-761ca79f0288" (UID: "ec23f89c-765f-4c30-a3e5-761ca79f0288"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:59:25.354257 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.349639 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec23f89c-765f-4c30-a3e5-761ca79f0288-kube-api-access-m95w2" pod "ec23f89c-765f-4c30-a3e5-761ca79f0288" (UID: "ec23f89c-765f-4c30-a3e5-761ca79f0288"). InnerVolumeSpecName "kube-api-access-m95w2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:59:25.354659 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.354594 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-istio-envoy" pod "ec23f89c-765f-4c30-a3e5-761ca79f0288" (UID: "ec23f89c-765f-4c30-a3e5-761ca79f0288"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:59:25.443117 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.443014 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec23f89c-765f-4c30-a3e5-761ca79f0288-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:59:25.443117 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.443054 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m95w2\" (UniqueName: \"kubernetes.io/projected/ec23f89c-765f-4c30-a3e5-761ca79f0288-kube-api-access-m95w2\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:59:25.443117 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.443071 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:59:25.443117 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.443085 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:59:25.443117 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.443098 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec23f89c-765f-4c30-a3e5-761ca79f0288-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:59:25.443117 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.443113 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec23f89c-765f-4c30-a3e5-761ca79f0288-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:59:25.443577 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.443126 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec23f89c-765f-4c30-a3e5-761ca79f0288-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 17:59:25.517024 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.516988 4471 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 17:59:25.517462 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.517392 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" podUID="c1e9cee8-e915-4a1f-819d-226dcda7485a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 17:59:25.654001 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.653935 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:59:25.654349 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.654285 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:59:25.654685 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.654557 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:59:25.736949 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.736852 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq"] Nov 02 17:59:25.838788 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.838762 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" event={"ID":"1ac349d9-ff43-493a-8cd2-3bfa0836a8da","Type":"ContainerStarted","Data":"1a7d11bdf91182fe9088444154d53e8748ca3da85b6f6f10c9726df6ddc3ffec"} Nov 02 17:59:25.840367 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.840341 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" Nov 02 17:59:25.840499 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.840344 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd" event={"ID":"ec23f89c-765f-4c30-a3e5-761ca79f0288","Type":"ContainerDied","Data":"6bdc92b4504236612ff16a808f9d4f0388f4dcbea09ba2443b37e7666abfc677"} Nov 02 17:59:25.840552 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.840511 4471 scope.go:122] "RemoveContainer" containerID="bc1f92a2701a88d7785bca48ccbed6f62a183cda50a47468e285924dc4e2a0ae" Nov 02 17:59:25.926752 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.926714 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd"] Nov 02 17:59:25.928356 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:25.928134 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-84kzd"] Nov 02 17:59:26.224275 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:26.224235 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 17:59:26.224766 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:26.224752 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 17:59:26.225686 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:26.225666 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 17:59:26.452131 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:26.452094 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec23f89c-765f-4c30-a3e5-761ca79f0288" path="/var/lib/kubelet/pods/ec23f89c-765f-4c30-a3e5-761ca79f0288/volumes" Nov 02 17:59:26.521249 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:26.517203 4471 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 17:59:26.521249 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:26.517314 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" podUID="c1e9cee8-e915-4a1f-819d-226dcda7485a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 17:59:26.850078 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:26.849979 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" event={"ID":"1ac349d9-ff43-493a-8cd2-3bfa0836a8da","Type":"ContainerStarted","Data":"2aef73ac8535c89039ffbaca42fd20b130d3319fc89de04458dad0d6e1c1172e"} Nov 02 17:59:26.913573 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:26.912405 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" podStartSLOduration=2.912391386 podStartE2EDuration="2.912391386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:26.911068292 +0000 UTC m=+1784.606423771" watchObservedRunningTime="2026-11-02 17:59:26.912391386 +0000 UTC m=+1784.607746859" Nov 02 17:59:27.172258 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:27.171757 4471 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 02 17:59:27.172258 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:27.171823 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" podUID="1ac349d9-ff43-493a-8cd2-3bfa0836a8da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 02 17:59:27.172258 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:27.172117 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 17:59:27.517649 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:27.517612 4471 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 17:59:27.517806 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:27.517684 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" podUID="c1e9cee8-e915-4a1f-819d-226dcda7485a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 17:59:28.163436 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:28.163399 4471 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 02 17:59:28.164057 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:28.164027 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" podUID="1ac349d9-ff43-493a-8cd2-3bfa0836a8da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 02 17:59:28.527339 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:28.527257 4471 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 17:59:28.527671 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:28.527627 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" podUID="c1e9cee8-e915-4a1f-819d-226dcda7485a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 17:59:28.745777 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:28.745743 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5fd3e0-77f74bdccf-xd8fb"] Nov 02 17:59:28.747904 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:28.746635 4471 cpu_manager.go:414] "containerMap: removing container" podUID="ec23f89c-765f-4c30-a3e5-761ca79f0288" containerName="istio-proxy" Nov 02 17:59:28.747904 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:28.746652 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec23f89c-765f-4c30-a3e5-761ca79f0288" containerName="istio-proxy" Nov 02 17:59:28.747904 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:28.746797 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec23f89c-765f-4c30-a3e5-761ca79f0288" containerName="istio-proxy" Nov 02 17:59:28.750802 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:28.750336 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5fd3e0-77f74bdccf-xd8fb" Nov 02 17:59:28.823081 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:28.823048 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5fd3e0\"" Nov 02 17:59:28.823634 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:28.823609 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5fd3e0-dockercfg-rbnss\"" Nov 02 17:59:28.826874 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:28.826848 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm"] Nov 02 17:59:28.832085 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:28.832062 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm" Nov 02 17:59:28.833427 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:28.833204 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5fd3e0-77f74bdccf-xd8fb"] Nov 02 17:59:28.841330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:28.841020 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm"] Nov 02 17:59:28.847807 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:28.847781 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5fd3e0-74cfc7bbf7-cc5bk"] Nov 02 17:59:28.853619 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:28.853575 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5fd3e0-74cfc7bbf7-cc5bk" Nov 02 17:59:28.910567 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:28.889448 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a29999\"/\"default-dockercfg-5ld8l\"" Nov 02 17:59:28.910567 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:28.889550 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a29999\"/\"kube-root-ca.crt\"" Nov 02 17:59:28.910567 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:28.889614 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a29999\"/\"openshift-service-ca.crt\"" Nov 02 17:59:28.910567 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:28.889706 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a29999\"/\"test-model-a29999-kserve-self-signed-certs\"" Nov 02 17:59:28.910857 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:28.910635 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5fd3e0-74cfc7bbf7-cc5bk"] Nov 02 17:59:28.922328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:28.920512 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctdlh\" (UniqueName: \"kubernetes.io/projected/a4d08194-c737-4a26-9198-ecfd8e6554f0-kube-api-access-ctdlh\") pod \"test-model-a29999-kserve-768f76f89c-67lgm\" (UID: \"a4d08194-c737-4a26-9198-ecfd8e6554f0\") " pod="ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm" Nov 02 17:59:28.922328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:28.920628 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7pgj\" (UniqueName: \"kubernetes.io/projected/d63f9cab-764f-46ae-a522-2ef02346c21a-kube-api-access-z7pgj\") pod \"payload-pre-processing-e2e-inf-b-5fd3e0-77f74bdccf-xd8fb\" (UID: \"d63f9cab-764f-46ae-a522-2ef02346c21a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5fd3e0-77f74bdccf-xd8fb" Nov 02 17:59:28.922328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:28.920798 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d63f9cab-764f-46ae-a522-2ef02346c21a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5fd3e0-77f74bdccf-xd8fb\" (UID: \"d63f9cab-764f-46ae-a522-2ef02346c21a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5fd3e0-77f74bdccf-xd8fb" Nov 02 17:59:28.922328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:28.920819 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4d08194-c737-4a26-9198-ecfd8e6554f0-home\") pod \"test-model-a29999-kserve-768f76f89c-67lgm\" (UID: \"a4d08194-c737-4a26-9198-ecfd8e6554f0\") " pod="ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm" Nov 02 17:59:28.922328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:28.920895 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4d08194-c737-4a26-9198-ecfd8e6554f0-tls-certs\") pod \"test-model-a29999-kserve-768f76f89c-67lgm\" (UID: \"a4d08194-c737-4a26-9198-ecfd8e6554f0\") " pod="ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm" Nov 02 17:59:28.922328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:28.920951 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4d08194-c737-4a26-9198-ecfd8e6554f0-tmp-dir\") pod \"test-model-a29999-kserve-768f76f89c-67lgm\" (UID: \"a4d08194-c737-4a26-9198-ecfd8e6554f0\") " pod="ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm" Nov 02 17:59:28.922328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:28.920985 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4d08194-c737-4a26-9198-ecfd8e6554f0-dshm\") pod \"test-model-a29999-kserve-768f76f89c-67lgm\" (UID: \"a4d08194-c737-4a26-9198-ecfd8e6554f0\") " pod="ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm" Nov 02 17:59:28.922328 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:28.921015 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4d08194-c737-4a26-9198-ecfd8e6554f0-model-cache\") pod \"test-model-a29999-kserve-768f76f89c-67lgm\" (UID: \"a4d08194-c737-4a26-9198-ecfd8e6554f0\") " pod="ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm" Nov 02 17:59:29.022299 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:29.022266 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctdlh\" (UniqueName: \"kubernetes.io/projected/a4d08194-c737-4a26-9198-ecfd8e6554f0-kube-api-access-ctdlh\") pod \"test-model-a29999-kserve-768f76f89c-67lgm\" (UID: \"a4d08194-c737-4a26-9198-ecfd8e6554f0\") " pod="ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm" Nov 02 17:59:29.022517 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:29.022414 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7wrk\" (UniqueName: \"kubernetes.io/projected/1c0b125a-116e-46b0-ad0b-0f7ff15fefa6-kube-api-access-l7wrk\") pod \"payload-processing-e2e-inf-b-5fd3e0-74cfc7bbf7-cc5bk\" (UID: \"1c0b125a-116e-46b0-ad0b-0f7ff15fefa6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5fd3e0-74cfc7bbf7-cc5bk" Nov 02 17:59:29.022517 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:29.022477 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7pgj\" (UniqueName: \"kubernetes.io/projected/d63f9cab-764f-46ae-a522-2ef02346c21a-kube-api-access-z7pgj\") pod \"payload-pre-processing-e2e-inf-b-5fd3e0-77f74bdccf-xd8fb\" (UID: \"d63f9cab-764f-46ae-a522-2ef02346c21a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5fd3e0-77f74bdccf-xd8fb" Nov 02 17:59:29.022908 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:29.022834 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d63f9cab-764f-46ae-a522-2ef02346c21a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5fd3e0-77f74bdccf-xd8fb\" (UID: \"d63f9cab-764f-46ae-a522-2ef02346c21a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5fd3e0-77f74bdccf-xd8fb" Nov 02 17:59:29.022908 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:29.022901 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4d08194-c737-4a26-9198-ecfd8e6554f0-home\") pod \"test-model-a29999-kserve-768f76f89c-67lgm\" (UID: \"a4d08194-c737-4a26-9198-ecfd8e6554f0\") " pod="ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm" Nov 02 17:59:29.022908 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:29.023018 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4d08194-c737-4a26-9198-ecfd8e6554f0-tls-certs\") pod \"test-model-a29999-kserve-768f76f89c-67lgm\" (UID: \"a4d08194-c737-4a26-9198-ecfd8e6554f0\") " pod="ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm" Nov 02 17:59:29.022908 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:29.023098 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4d08194-c737-4a26-9198-ecfd8e6554f0-tmp-dir\") pod \"test-model-a29999-kserve-768f76f89c-67lgm\" (UID: \"a4d08194-c737-4a26-9198-ecfd8e6554f0\") " pod="ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm" Nov 02 17:59:29.022908 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:29.023205 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4d08194-c737-4a26-9198-ecfd8e6554f0-dshm\") pod \"test-model-a29999-kserve-768f76f89c-67lgm\" (UID: \"a4d08194-c737-4a26-9198-ecfd8e6554f0\") " pod="ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm" Nov 02 17:59:29.022908 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:29.023277 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0b125a-116e-46b0-ad0b-0f7ff15fefa6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5fd3e0-74cfc7bbf7-cc5bk\" (UID: \"1c0b125a-116e-46b0-ad0b-0f7ff15fefa6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5fd3e0-74cfc7bbf7-cc5bk" Nov 02 17:59:29.022908 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:29.023359 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4d08194-c737-4a26-9198-ecfd8e6554f0-model-cache\") pod \"test-model-a29999-kserve-768f76f89c-67lgm\" (UID: \"a4d08194-c737-4a26-9198-ecfd8e6554f0\") " pod="ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm" Nov 02 17:59:29.022908 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:29.023929 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d63f9cab-764f-46ae-a522-2ef02346c21a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5fd3e0-77f74bdccf-xd8fb\" (UID: \"d63f9cab-764f-46ae-a522-2ef02346c21a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5fd3e0-77f74bdccf-xd8fb" Nov 02 17:59:29.024811 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:29.024508 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4d08194-c737-4a26-9198-ecfd8e6554f0-tmp-dir\") pod \"test-model-a29999-kserve-768f76f89c-67lgm\" (UID: \"a4d08194-c737-4a26-9198-ecfd8e6554f0\") " pod="ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm" Nov 02 17:59:29.024811 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:29.024778 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4d08194-c737-4a26-9198-ecfd8e6554f0-home\") pod \"test-model-a29999-kserve-768f76f89c-67lgm\" (UID: \"a4d08194-c737-4a26-9198-ecfd8e6554f0\") " pod="ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm" Nov 02 17:59:29.025319 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:29.025127 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4d08194-c737-4a26-9198-ecfd8e6554f0-model-cache\") pod \"test-model-a29999-kserve-768f76f89c-67lgm\" (UID: \"a4d08194-c737-4a26-9198-ecfd8e6554f0\") " pod="ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm" Nov 02 17:59:29.028628 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:29.028607 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4d08194-c737-4a26-9198-ecfd8e6554f0-dshm\") pod \"test-model-a29999-kserve-768f76f89c-67lgm\" (UID: \"a4d08194-c737-4a26-9198-ecfd8e6554f0\") " pod="ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm" Nov 02 17:59:29.029743 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:29.029699 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4d08194-c737-4a26-9198-ecfd8e6554f0-tls-certs\") pod \"test-model-a29999-kserve-768f76f89c-67lgm\" (UID: \"a4d08194-c737-4a26-9198-ecfd8e6554f0\") " pod="ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm" Nov 02 17:59:29.071381 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:29.049152 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7pgj\" (UniqueName: \"kubernetes.io/projected/d63f9cab-764f-46ae-a522-2ef02346c21a-kube-api-access-z7pgj\") pod \"payload-pre-processing-e2e-inf-b-5fd3e0-77f74bdccf-xd8fb\" (UID: \"d63f9cab-764f-46ae-a522-2ef02346c21a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5fd3e0-77f74bdccf-xd8fb" Nov 02 17:59:29.071381 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:29.055945 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctdlh\" (UniqueName: \"kubernetes.io/projected/a4d08194-c737-4a26-9198-ecfd8e6554f0-kube-api-access-ctdlh\") pod \"test-model-a29999-kserve-768f76f89c-67lgm\" (UID: \"a4d08194-c737-4a26-9198-ecfd8e6554f0\") " pod="ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm" Nov 02 17:59:29.124111 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:29.124068 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5fd3e0-77f74bdccf-xd8fb" Nov 02 17:59:29.129258 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:29.127375 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0b125a-116e-46b0-ad0b-0f7ff15fefa6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5fd3e0-74cfc7bbf7-cc5bk\" (UID: \"1c0b125a-116e-46b0-ad0b-0f7ff15fefa6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5fd3e0-74cfc7bbf7-cc5bk" Nov 02 17:59:29.129575 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:29.128274 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0b125a-116e-46b0-ad0b-0f7ff15fefa6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5fd3e0-74cfc7bbf7-cc5bk\" (UID: \"1c0b125a-116e-46b0-ad0b-0f7ff15fefa6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5fd3e0-74cfc7bbf7-cc5bk" Nov 02 17:59:29.144327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:29.130121 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7wrk\" (UniqueName: \"kubernetes.io/projected/1c0b125a-116e-46b0-ad0b-0f7ff15fefa6-kube-api-access-l7wrk\") pod \"payload-processing-e2e-inf-b-5fd3e0-74cfc7bbf7-cc5bk\" (UID: \"1c0b125a-116e-46b0-ad0b-0f7ff15fefa6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5fd3e0-74cfc7bbf7-cc5bk" Nov 02 17:59:29.213699 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:29.213657 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm" Nov 02 17:59:29.214941 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:29.214912 4471 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 02 17:59:29.215369 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:29.215137 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" podUID="1ac349d9-ff43-493a-8cd2-3bfa0836a8da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 02 17:59:29.227689 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:29.227649 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7wrk\" (UniqueName: \"kubernetes.io/projected/1c0b125a-116e-46b0-ad0b-0f7ff15fefa6-kube-api-access-l7wrk\") pod \"payload-processing-e2e-inf-b-5fd3e0-74cfc7bbf7-cc5bk\" (UID: \"1c0b125a-116e-46b0-ad0b-0f7ff15fefa6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5fd3e0-74cfc7bbf7-cc5bk" Nov 02 17:59:29.228155 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:29.228136 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5fd3e0-74cfc7bbf7-cc5bk" Nov 02 17:59:29.517012 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:29.516977 4471 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 17:59:29.517274 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:29.517240 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" podUID="c1e9cee8-e915-4a1f-819d-226dcda7485a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 17:59:30.191569 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:30.164706 4471 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 02 17:59:30.191569 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:30.164787 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" podUID="1ac349d9-ff43-493a-8cd2-3bfa0836a8da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 02 17:59:30.197423 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:30.197400 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5fd3e0-77f74bdccf-xd8fb"] Nov 02 17:59:30.204692 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:30.204654 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm"] Nov 02 17:59:30.211103 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:59:30.207771 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c0b125a_116e_46b0_ad0b_0f7ff15fefa6.slice/crio-0e689da343c04125d352d43ed448689fb08caf0f0ca0fe7cb7536419438e287c WatchSource:0}: Error finding container 0e689da343c04125d352d43ed448689fb08caf0f0ca0fe7cb7536419438e287c: Status 404 returned error can't find the container with id 0e689da343c04125d352d43ed448689fb08caf0f0ca0fe7cb7536419438e287c Nov 02 17:59:30.213124 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:30.213081 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5fd3e0-74cfc7bbf7-cc5bk"] Nov 02 17:59:30.522509 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:30.522479 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 17:59:30.523006 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:30.522943 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 17:59:30.523758 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:30.523735 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 17:59:30.940800 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:30.940717 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5fd3e0-74cfc7bbf7-cc5bk" event={"ID":"1c0b125a-116e-46b0-ad0b-0f7ff15fefa6","Type":"ContainerStarted","Data":"6127dcc0d44863ba678229d696cb6036d9d70b5ce71298022129cca0cbdb47bc"} Nov 02 17:59:30.940800 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:30.940774 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5fd3e0-74cfc7bbf7-cc5bk" event={"ID":"1c0b125a-116e-46b0-ad0b-0f7ff15fefa6","Type":"ContainerStarted","Data":"0e689da343c04125d352d43ed448689fb08caf0f0ca0fe7cb7536419438e287c"} Nov 02 17:59:30.940984 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:30.940807 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5fd3e0-74cfc7bbf7-cc5bk" Nov 02 17:59:30.942518 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:30.942491 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5fd3e0-77f74bdccf-xd8fb" event={"ID":"d63f9cab-764f-46ae-a522-2ef02346c21a","Type":"ContainerStarted","Data":"86d37e9151ad330bee33c9c89ebeeb0033c7876b4e997a9b89b3a3355eda179d"} Nov 02 17:59:30.942670 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:30.942537 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5fd3e0-77f74bdccf-xd8fb" event={"ID":"d63f9cab-764f-46ae-a522-2ef02346c21a","Type":"ContainerStarted","Data":"b744e7e3952ec1ae88222ae6ab290caa3b85c4bb7cf6f9af0f4d9380a29e5168"} Nov 02 17:59:30.942670 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:30.942582 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5fd3e0-77f74bdccf-xd8fb" Nov 02 17:59:30.944279 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:30.944254 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm" event={"ID":"a4d08194-c737-4a26-9198-ecfd8e6554f0","Type":"ContainerStarted","Data":"89cd0366ba9b02dc7892197fca2c1426b66ef294d90b91a736cffb13bc3932ff"} Nov 02 17:59:30.944389 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:30.944287 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm" event={"ID":"a4d08194-c737-4a26-9198-ecfd8e6554f0","Type":"ContainerStarted","Data":"603b36bfb44d77c8146a2fdeac837d77ed75ac0b254f485a012d43b50d2b6272"} Nov 02 17:59:30.990544 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:30.990475 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-5fd3e0-74cfc7bbf7-cc5bk" podStartSLOduration=2.990458701 podStartE2EDuration="2.990458701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:30.986891029 +0000 UTC m=+1788.682246526" watchObservedRunningTime="2026-11-02 17:59:30.990458701 +0000 UTC m=+1788.685814228" Nov 02 17:59:31.042978 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:31.042919 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm" podStartSLOduration=3.0428994 podStartE2EDuration="3.0428994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:31.037678015 +0000 UTC m=+1788.733033516" watchObservedRunningTime="2026-11-02 17:59:31.0428994 +0000 UTC m=+1788.738254870" Nov 02 17:59:31.113710 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:31.113657 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5fd3e0-77f74bdccf-xd8fb" podStartSLOduration=3.1136417769999998 podStartE2EDuration="3.113641777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:31.111640827 +0000 UTC m=+1788.806996301" watchObservedRunningTime="2026-11-02 17:59:31.113641777 +0000 UTC m=+1788.808997251" Nov 02 17:59:31.180416 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:31.180364 4471 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 02 17:59:31.180615 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:31.180442 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" podUID="1ac349d9-ff43-493a-8cd2-3bfa0836a8da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 02 17:59:31.215741 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:31.215661 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm" Nov 02 17:59:32.180371 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:32.176723 4471 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 02 17:59:32.180371 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:32.176792 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" podUID="1ac349d9-ff43-493a-8cd2-3bfa0836a8da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 02 17:59:32.978576 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:32.978541 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn"] Nov 02 17:59:32.982277 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:32.982259 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 17:59:33.013808 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.013752 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn"] Nov 02 17:59:33.029359 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.028098 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9"] Nov 02 17:59:33.037761 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.037729 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 17:59:33.055448 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.055398 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-credential-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 17:59:33.055794 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.055762 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-workload-certs\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 17:59:33.056037 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.055999 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istio-podinfo\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 17:59:33.056146 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.056064 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istio-envoy\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 17:59:33.056250 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.056155 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-workload-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 17:59:33.056250 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.056201 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istio-token\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 17:59:33.056786 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.056399 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istio-data\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 17:59:33.056786 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.056601 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzxwm\" (UniqueName: \"kubernetes.io/projected/e881e4aa-a5ad-46d2-952d-97c5118e98b3-kube-api-access-nzxwm\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 17:59:33.056786 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.056653 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istiod-ca-cert\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 17:59:33.057411 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.057386 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9"] Nov 02 17:59:33.157947 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.157909 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-workload-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 17:59:33.158126 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.157971 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istio-data\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 17:59:33.158126 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.158051 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38fea887-fcc0-42fa-a1c9-320e44f70562-istiod-ca-cert\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 17:59:33.158126 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.158103 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75csh\" (UniqueName: \"kubernetes.io/projected/38fea887-fcc0-42fa-a1c9-320e44f70562-kube-api-access-75csh\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 17:59:33.158358 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.158289 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzxwm\" (UniqueName: \"kubernetes.io/projected/e881e4aa-a5ad-46d2-952d-97c5118e98b3-kube-api-access-nzxwm\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 17:59:33.158440 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.158400 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-credential-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 17:59:33.158440 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.158432 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istiod-ca-cert\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 17:59:33.158520 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.158444 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istio-data\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 17:59:33.158520 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.158488 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-credential-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 17:59:33.158601 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.158544 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-istio-envoy\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 17:59:33.158652 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.158638 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-workload-certs\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 17:59:33.158707 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.158684 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-workload-certs\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 17:59:33.158762 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.158716 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38fea887-fcc0-42fa-a1c9-320e44f70562-istio-token\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 17:59:33.158823 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.158771 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istio-podinfo\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 17:59:33.158823 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.158800 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-istio-data\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 17:59:33.158936 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.158834 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-credential-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 17:59:33.158936 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.158842 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istio-envoy\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 17:59:33.158936 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.158925 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38fea887-fcc0-42fa-a1c9-320e44f70562-istio-podinfo\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 17:59:33.159083 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.158999 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-workload-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 17:59:33.159083 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.159041 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-workload-certs\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 17:59:33.159083 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.159046 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istio-token\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 17:59:33.159236 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.159217 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istiod-ca-cert\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 17:59:33.159419 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.159395 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-workload-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 17:59:33.161398 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.161373 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istio-envoy\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 17:59:33.161610 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.161591 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istio-podinfo\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 17:59:33.166700 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.166680 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 17:59:33.167148 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.167129 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 17:59:33.167972 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.167952 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 17:59:33.187900 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.187871 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzxwm\" (UniqueName: \"kubernetes.io/projected/e881e4aa-a5ad-46d2-952d-97c5118e98b3-kube-api-access-nzxwm\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 17:59:33.188182 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.187995 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istio-token\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 17:59:33.260850 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.260753 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-workload-certs\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 17:59:33.260850 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.260792 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38fea887-fcc0-42fa-a1c9-320e44f70562-istio-token\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 17:59:33.260850 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.260838 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-istio-data\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 17:59:33.261146 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.260884 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38fea887-fcc0-42fa-a1c9-320e44f70562-istio-podinfo\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 17:59:33.261146 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.261057 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-workload-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 17:59:33.261146 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.261134 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38fea887-fcc0-42fa-a1c9-320e44f70562-istiod-ca-cert\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 17:59:33.261336 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.261247 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-workload-certs\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 17:59:33.261336 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.261275 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-istio-data\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 17:59:33.261336 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.261274 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75csh\" (UniqueName: \"kubernetes.io/projected/38fea887-fcc0-42fa-a1c9-320e44f70562-kube-api-access-75csh\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 17:59:33.261514 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.261491 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-credential-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 17:59:33.261576 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.261523 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-workload-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 17:59:33.261640 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.261610 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-istio-envoy\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 17:59:33.261847 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.261823 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-credential-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 17:59:33.261908 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.261846 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38fea887-fcc0-42fa-a1c9-320e44f70562-istiod-ca-cert\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 17:59:33.263572 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.263548 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38fea887-fcc0-42fa-a1c9-320e44f70562-istio-podinfo\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 17:59:33.263673 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.263662 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-istio-envoy\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 17:59:33.293088 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.293055 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75csh\" (UniqueName: \"kubernetes.io/projected/38fea887-fcc0-42fa-a1c9-320e44f70562-kube-api-access-75csh\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 17:59:33.298683 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.298647 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38fea887-fcc0-42fa-a1c9-320e44f70562-istio-token\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 17:59:33.298996 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.298973 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 17:59:33.363889 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.363853 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 17:59:33.793358 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:59:33.793122 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode881e4aa_a5ad_46d2_952d_97c5118e98b3.slice/crio-f2d23fa64798b2c0c58eb3c4a6b293d13c37f438ed2462d2d3bd111968822bda WatchSource:0}: Error finding container f2d23fa64798b2c0c58eb3c4a6b293d13c37f438ed2462d2d3bd111968822bda: Status 404 returned error can't find the container with id f2d23fa64798b2c0c58eb3c4a6b293d13c37f438ed2462d2d3bd111968822bda Nov 02 17:59:33.798629 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.798565 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:59:33.799352 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.799328 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:59:33.799544 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.799525 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:59:33.819982 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:33.803228 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn"] Nov 02 17:59:34.057342 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:34.055280 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" event={"ID":"e881e4aa-a5ad-46d2-952d-97c5118e98b3","Type":"ContainerStarted","Data":"8563149652f5df2a3f53cb8f26fd0d8520a8dcd7d6ce35447e9870c5fcd40331"} Nov 02 17:59:34.057342 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:34.055366 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" event={"ID":"e881e4aa-a5ad-46d2-952d-97c5118e98b3","Type":"ContainerStarted","Data":"f2d23fa64798b2c0c58eb3c4a6b293d13c37f438ed2462d2d3bd111968822bda"} Nov 02 17:59:34.113205 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:34.113146 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" podStartSLOduration=2.113130358 podStartE2EDuration="2.113130358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:34.110771232 +0000 UTC m=+1791.806126726" watchObservedRunningTime="2026-11-02 17:59:34.113130358 +0000 UTC m=+1791.808485832" Nov 02 17:59:34.134887 ip-10-0-142-26 kubenswrapper[4471]: W1102 17:59:34.134829 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38fea887_fcc0_42fa_a1c9_320e44f70562.slice/crio-11a5437f8ec75661c50f6b6373cad7035321997d4f1e1e964da881b43434d8b0 WatchSource:0}: Error finding container 11a5437f8ec75661c50f6b6373cad7035321997d4f1e1e964da881b43434d8b0: Status 404 returned error can't find the container with id 11a5437f8ec75661c50f6b6373cad7035321997d4f1e1e964da881b43434d8b0 Nov 02 17:59:34.138589 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:34.138547 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9"] Nov 02 17:59:34.138979 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:34.138946 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:59:34.139025 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:34.139016 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:59:34.139058 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:34.139045 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:59:34.299359 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:34.299321 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 17:59:34.300780 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:34.300749 4471 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 02 17:59:34.300907 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:34.300806 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" podUID="e881e4aa-a5ad-46d2-952d-97c5118e98b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 02 17:59:35.062844 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:35.062803 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" event={"ID":"38fea887-fcc0-42fa-a1c9-320e44f70562","Type":"ContainerStarted","Data":"2f33607f9e421093ec91c99de5a98130292399afa5ee9d6fa12df8e0e7e23650"} Nov 02 17:59:35.063040 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:35.062898 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" event={"ID":"38fea887-fcc0-42fa-a1c9-320e44f70562","Type":"ContainerStarted","Data":"11a5437f8ec75661c50f6b6373cad7035321997d4f1e1e964da881b43434d8b0"} Nov 02 17:59:35.129979 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:35.129929 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" podStartSLOduration=3.129913257 podStartE2EDuration="3.129913257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:35.1249811 +0000 UTC m=+1792.820336605" watchObservedRunningTime="2026-11-02 17:59:35.129913257 +0000 UTC m=+1792.825268730" Nov 02 17:59:35.300598 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:35.300556 4471 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 02 17:59:35.301207 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:35.300611 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" podUID="e881e4aa-a5ad-46d2-952d-97c5118e98b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 02 17:59:35.371031 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:35.370894 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 17:59:35.397495 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:35.397449 4471 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 02 17:59:35.400138 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:35.400097 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" podUID="38fea887-fcc0-42fa-a1c9-320e44f70562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 02 17:59:36.310498 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:36.310451 4471 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 02 17:59:36.311018 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:36.310518 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" podUID="e881e4aa-a5ad-46d2-952d-97c5118e98b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 02 17:59:36.371331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:36.370977 4471 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 02 17:59:36.371331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:36.371045 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" podUID="38fea887-fcc0-42fa-a1c9-320e44f70562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 02 17:59:37.235706 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.235672 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4"] Nov 02 17:59:37.241462 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.241438 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 17:59:37.312737 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.312699 4471 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 02 17:59:37.313862 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.313825 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" podUID="e881e4aa-a5ad-46d2-952d-97c5118e98b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 02 17:59:37.325097 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.320776 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-workload-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 17:59:37.325986 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.325956 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-istio-envoy\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 17:59:37.326550 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.326293 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790f4f11-1fb0-49ae-871d-07295b20a266-istiod-ca-cert\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 17:59:37.326798 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.326778 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-workload-certs\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 17:59:37.327004 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.326985 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-credential-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 17:59:37.327255 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.327236 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-istio-data\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 17:59:37.327510 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.327489 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb6z7\" (UniqueName: \"kubernetes.io/projected/790f4f11-1fb0-49ae-871d-07295b20a266-kube-api-access-vb6z7\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 17:59:37.327763 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.327739 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790f4f11-1fb0-49ae-871d-07295b20a266-istio-podinfo\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 17:59:37.331610 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.331590 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/790f4f11-1fb0-49ae-871d-07295b20a266-istio-token\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 17:59:37.331884 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.331352 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4"] Nov 02 17:59:37.340348 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.338389 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c"] Nov 02 17:59:37.343190 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.343172 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 17:59:37.363373 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.355084 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c"] Nov 02 17:59:37.588398 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.540469 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790f4f11-1fb0-49ae-871d-07295b20a266-istio-podinfo\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 17:59:37.588398 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.540523 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/790f4f11-1fb0-49ae-871d-07295b20a266-istio-token\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 17:59:37.588398 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.540704 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-workload-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 17:59:37.588398 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.541265 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-workload-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 17:59:37.588398 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.542200 4471 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 02 17:59:37.588398 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.542282 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" podUID="38fea887-fcc0-42fa-a1c9-320e44f70562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 02 17:59:37.588398 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.543123 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-istio-envoy\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 17:59:37.588398 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.543244 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790f4f11-1fb0-49ae-871d-07295b20a266-istiod-ca-cert\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 17:59:37.588398 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.543381 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-workload-certs\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 17:59:37.588398 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.543467 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-credential-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 17:59:37.588398 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.543569 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-istio-data\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 17:59:37.588398 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.543636 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb6z7\" (UniqueName: \"kubernetes.io/projected/790f4f11-1fb0-49ae-871d-07295b20a266-kube-api-access-vb6z7\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 17:59:37.588398 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.563218 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-workload-certs\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 17:59:37.631069 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.631032 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-istio-data\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 17:59:37.631356 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.631334 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790f4f11-1fb0-49ae-871d-07295b20a266-istiod-ca-cert\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 17:59:37.643717 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.643679 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-credential-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 17:59:37.654879 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.654820 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb6z7\" (UniqueName: \"kubernetes.io/projected/790f4f11-1fb0-49ae-871d-07295b20a266-kube-api-access-vb6z7\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 17:59:37.692703 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.655508 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790f4f11-1fb0-49ae-871d-07295b20a266-istio-podinfo\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 17:59:37.692993 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.655911 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/790f4f11-1fb0-49ae-871d-07295b20a266-istio-token\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 17:59:37.693178 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.675261 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-istio-envoy\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 17:59:37.693290 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.693035 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-credential-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 17:59:37.693469 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.693451 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rgpz\" (UniqueName: \"kubernetes.io/projected/aaca053e-d726-46af-a659-2a15071343c7-kube-api-access-6rgpz\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 17:59:37.693627 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.693610 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaca053e-d726-46af-a659-2a15071343c7-istiod-ca-cert\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 17:59:37.693809 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.693789 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-istio-envoy\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 17:59:37.693979 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.693963 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-workload-certs\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 17:59:37.694207 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.694191 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaca053e-d726-46af-a659-2a15071343c7-istio-podinfo\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 17:59:37.694380 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.694363 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-istio-data\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 17:59:37.694665 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.694638 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaca053e-d726-46af-a659-2a15071343c7-istio-token\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 17:59:37.694955 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.694934 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-workload-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 17:59:37.811469 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.806980 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-workload-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 17:59:37.811469 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.807061 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-credential-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 17:59:37.811469 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.807110 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rgpz\" (UniqueName: \"kubernetes.io/projected/aaca053e-d726-46af-a659-2a15071343c7-kube-api-access-6rgpz\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 17:59:37.811469 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.807160 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaca053e-d726-46af-a659-2a15071343c7-istiod-ca-cert\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 17:59:37.811469 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.807232 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-istio-envoy\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 17:59:37.811469 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.807286 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-workload-certs\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 17:59:37.811469 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.807416 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaca053e-d726-46af-a659-2a15071343c7-istio-podinfo\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 17:59:37.811469 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.807479 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-istio-data\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 17:59:37.811469 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.807624 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaca053e-d726-46af-a659-2a15071343c7-istio-token\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 17:59:37.851596 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.850003 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-credential-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 17:59:37.851596 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.850726 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaca053e-d726-46af-a659-2a15071343c7-istiod-ca-cert\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 17:59:37.851596 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.851497 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-workload-certs\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 17:59:37.852167 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.852120 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-istio-data\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 17:59:37.852756 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.852733 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-workload-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 17:59:37.881510 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.880272 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaca053e-d726-46af-a659-2a15071343c7-istio-token\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 17:59:37.889330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.884851 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaca053e-d726-46af-a659-2a15071343c7-istio-podinfo\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 17:59:37.917704 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.910027 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rgpz\" (UniqueName: \"kubernetes.io/projected/aaca053e-d726-46af-a659-2a15071343c7-kube-api-access-6rgpz\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 17:59:37.923636 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.923602 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-istio-envoy\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 17:59:37.925220 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.924975 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 17:59:37.925613 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:37.925591 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 17:59:38.357199 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:38.357127 4471 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 02 17:59:38.359565 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:38.359526 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" podUID="e881e4aa-a5ad-46d2-952d-97c5118e98b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 02 17:59:38.365971 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:38.365938 4471 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 02 17:59:38.366205 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:38.366177 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" podUID="38fea887-fcc0-42fa-a1c9-320e44f70562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 02 17:59:38.476403 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:38.470455 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:59:38.544438 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:38.538147 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" podUID="c3e6eb3c-0496-44c8-a45f-9b4c662756b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.155:8000/ready\": context deadline exceeded" Nov 02 17:59:38.545598 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:38.545557 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:59:38.668548 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:38.664060 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:59:38.668548 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:38.664290 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:59:39.219492 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:39.216082 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm" Nov 02 17:59:39.315878 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:39.315837 4471 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 02 17:59:39.316166 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:39.316097 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" podUID="e881e4aa-a5ad-46d2-952d-97c5118e98b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 02 17:59:39.333402 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:39.331861 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm" Nov 02 17:59:39.334863 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:39.334821 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:59:39.334945 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:39.334914 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:59:39.334980 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:39.334956 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:59:39.336162 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:39.336137 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4"] Nov 02 17:59:39.418607 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:39.418575 4471 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 02 17:59:39.419532 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:39.419122 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" podUID="38fea887-fcc0-42fa-a1c9-320e44f70562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 02 17:59:39.429713 ip-10-0-142-26 kubenswrapper[4471]: E1102 17:59:39.427864 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 17:59:39.438731 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:39.438329 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" event={"ID":"790f4f11-1fb0-49ae-871d-07295b20a266","Type":"ContainerStarted","Data":"4b73a36db6ac3328a497cacce764a8e6a7dcd9f3272e02b4e80004a28cb563b8"} Nov 02 17:59:39.453367 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:39.453339 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm" Nov 02 17:59:39.545094 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:39.528757 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c"] Nov 02 17:59:39.549410 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:39.549360 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:59:39.549541 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:39.549518 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:59:39.549635 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:39.549612 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:59:40.319330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:40.300833 4471 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 02 17:59:40.319330 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:40.300905 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" podUID="e881e4aa-a5ad-46d2-952d-97c5118e98b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 02 17:59:40.411881 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:40.411841 4471 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 02 17:59:40.412128 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:40.412096 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" podUID="38fea887-fcc0-42fa-a1c9-320e44f70562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 02 17:59:40.463329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:40.455275 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" event={"ID":"aaca053e-d726-46af-a659-2a15071343c7","Type":"ContainerStarted","Data":"d0b8e1501153337b07e1139059c008fa0659d99849d80e336abecb53916df430"} Nov 02 17:59:40.463329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:40.455400 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" event={"ID":"aaca053e-d726-46af-a659-2a15071343c7","Type":"ContainerStarted","Data":"737fc06c97e32d47f4f48b2b15ea6bd3c10d4160b079d95343fa616768f472fa"} Nov 02 17:59:40.531332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:40.530871 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" event={"ID":"790f4f11-1fb0-49ae-871d-07295b20a266","Type":"ContainerStarted","Data":"b9e5afb1a653ef91c046b75fb888f50e24ac03adff234fed4762cbaab3baaf7f"} Nov 02 17:59:40.626236 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:40.626140 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" podStartSLOduration=3.626123672 podStartE2EDuration="3.626123672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:40.554435724 +0000 UTC m=+1798.249791235" watchObservedRunningTime="2026-11-02 17:59:40.626123672 +0000 UTC m=+1798.321479144" Nov 02 17:59:40.631998 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:40.631940 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" podStartSLOduration=3.63190745 podStartE2EDuration="3.63190745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:40.620838926 +0000 UTC m=+1798.316194420" watchObservedRunningTime="2026-11-02 17:59:40.63190745 +0000 UTC m=+1798.327262925" Nov 02 17:59:40.926648 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:40.926566 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 17:59:40.926648 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:40.926629 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 17:59:40.927563 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:40.927536 4471 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 02 17:59:40.927563 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:40.927553 4471 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Nov 02 17:59:40.927713 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:40.927578 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" podUID="aaca053e-d726-46af-a659-2a15071343c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 02 17:59:40.927713 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:40.927593 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" podUID="790f4f11-1fb0-49ae-871d-07295b20a266" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Nov 02 17:59:41.301332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:41.301189 4471 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 02 17:59:41.301332 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:41.301254 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" podUID="e881e4aa-a5ad-46d2-952d-97c5118e98b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 02 17:59:41.365327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:41.364768 4471 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 02 17:59:41.365327 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:41.364829 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" podUID="38fea887-fcc0-42fa-a1c9-320e44f70562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 02 17:59:41.963344 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:41.938451 4471 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 02 17:59:41.963344 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:41.938537 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" podUID="aaca053e-d726-46af-a659-2a15071343c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 02 17:59:41.963344 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:41.938985 4471 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Nov 02 17:59:41.963344 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:41.939024 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" podUID="790f4f11-1fb0-49ae-871d-07295b20a266" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Nov 02 17:59:42.023331 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:42.020152 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5fd3e0-74cfc7bbf7-cc5bk" Nov 02 17:59:42.024683 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:42.023656 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5fd3e0-77f74bdccf-xd8fb" Nov 02 17:59:42.397329 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:42.359603 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 17:59:42.429335 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:42.423779 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 17:59:42.429335 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:42.427429 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 17:59:42.429335 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:42.427826 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 17:59:42.533114 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:42.526316 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 17:59:42.533114 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:42.526410 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 17:59:42.932281 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:42.932233 4471 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 02 17:59:42.932553 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:42.932284 4471 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Nov 02 17:59:42.932553 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:42.932320 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" podUID="aaca053e-d726-46af-a659-2a15071343c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 02 17:59:42.932553 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:42.932354 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" podUID="790f4f11-1fb0-49ae-871d-07295b20a266" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Nov 02 17:59:43.934132 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:43.933653 4471 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Nov 02 17:59:43.934132 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:43.933714 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" podUID="790f4f11-1fb0-49ae-871d-07295b20a266" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Nov 02 17:59:43.934848 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:43.934810 4471 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 02 17:59:43.935018 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:43.934994 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" podUID="aaca053e-d726-46af-a659-2a15071343c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 02 17:59:44.927324 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:44.927277 4471 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Nov 02 17:59:44.927499 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:44.927296 4471 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 02 17:59:44.927499 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:44.927361 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" podUID="790f4f11-1fb0-49ae-871d-07295b20a266" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Nov 02 17:59:44.927499 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:44.927365 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" podUID="aaca053e-d726-46af-a659-2a15071343c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 02 17:59:45.926557 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:45.926514 4471 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 02 17:59:45.926931 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:45.926579 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" podUID="aaca053e-d726-46af-a659-2a15071343c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 02 17:59:45.926931 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:45.926514 4471 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Nov 02 17:59:45.926931 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:45.926694 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" podUID="790f4f11-1fb0-49ae-871d-07295b20a266" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Nov 02 17:59:46.926934 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:46.926889 4471 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 02 17:59:46.927473 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:46.926974 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" podUID="aaca053e-d726-46af-a659-2a15071343c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 02 17:59:46.927473 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:46.927053 4471 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Nov 02 17:59:46.927473 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:46.927105 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" podUID="790f4f11-1fb0-49ae-871d-07295b20a266" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Nov 02 17:59:47.931630 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:47.931600 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 17:59:47.932084 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:47.931673 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 17:59:47.932386 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:47.932362 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 17:59:47.932570 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:47.932550 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 17:59:47.932887 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:47.932865 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 17:59:47.933083 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:47.933066 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 17:59:59.071018 ip-10-0-142-26 kubenswrapper[4471]: I1102 17:59:59.070966 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:00:00.258398 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:00.258359 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr"] Nov 02 18:00:00.286567 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:00.286535 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr" Nov 02 18:00:00.335270 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:00.335230 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2"] Nov 02 18:00:00.344529 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:00.344498 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9"] Nov 02 18:00:00.363339 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:00.360570 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 18:00:00.383334 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:00.370219 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2" Nov 02 18:00:00.384853 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:00.384814 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9" Nov 02 18:00:00.425328 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:00.424947 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr\" (UID: \"16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr" Nov 02 18:00:00.425328 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:00.425260 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4k8m\" (UniqueName: \"kubernetes.io/projected/16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4-kube-api-access-v4k8m\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr\" (UID: \"16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr" Nov 02 18:00:00.435932 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:00.425613 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr"] Nov 02 18:00:00.722373 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:00.670748 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpbkx\" (UniqueName: \"kubernetes.io/projected/fab397a0-94f3-4ac9-9eea-36e8ca65ce1e-kube-api-access-fpbkx\") pod \"maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2\" (UID: \"fab397a0-94f3-4ac9-9eea-36e8ca65ce1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2" Nov 02 18:00:00.727496 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:00.727453 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8svz9\" (UniqueName: \"kubernetes.io/projected/9206222b-4b98-4c20-b48c-dc6d671948c8-kube-api-access-8svz9\") pod \"maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9\" (UID: \"9206222b-4b98-4c20-b48c-dc6d671948c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9" Nov 02 18:00:00.727787 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:00.727763 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fab397a0-94f3-4ac9-9eea-36e8ca65ce1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2\" (UID: \"fab397a0-94f3-4ac9-9eea-36e8ca65ce1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2" Nov 02 18:00:00.728062 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:00.728041 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4k8m\" (UniqueName: \"kubernetes.io/projected/16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4-kube-api-access-v4k8m\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr\" (UID: \"16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr" Nov 02 18:00:00.728479 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:00.728453 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9206222b-4b98-4c20-b48c-dc6d671948c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9\" (UID: \"9206222b-4b98-4c20-b48c-dc6d671948c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9" Nov 02 18:00:00.846388 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:00.834438 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr\" (UID: \"16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr" Nov 02 18:00:00.846388 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:00.835515 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr\" (UID: \"16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr" Nov 02 18:00:00.988661 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:00.950147 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpbkx\" (UniqueName: \"kubernetes.io/projected/fab397a0-94f3-4ac9-9eea-36e8ca65ce1e-kube-api-access-fpbkx\") pod \"maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2\" (UID: \"fab397a0-94f3-4ac9-9eea-36e8ca65ce1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2" Nov 02 18:00:00.989467 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:00.989437 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8svz9\" (UniqueName: \"kubernetes.io/projected/9206222b-4b98-4c20-b48c-dc6d671948c8-kube-api-access-8svz9\") pod \"maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9\" (UID: \"9206222b-4b98-4c20-b48c-dc6d671948c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9" Nov 02 18:00:00.989753 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:00.989733 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fab397a0-94f3-4ac9-9eea-36e8ca65ce1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2\" (UID: \"fab397a0-94f3-4ac9-9eea-36e8ca65ce1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2" Nov 02 18:00:00.990318 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:00.990277 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9206222b-4b98-4c20-b48c-dc6d671948c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9\" (UID: \"9206222b-4b98-4c20-b48c-dc6d671948c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9" Nov 02 18:00:01.005483 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:00.992138 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fab397a0-94f3-4ac9-9eea-36e8ca65ce1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2\" (UID: \"fab397a0-94f3-4ac9-9eea-36e8ca65ce1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2" Nov 02 18:00:01.005483 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:00.982376 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4k8m\" (UniqueName: \"kubernetes.io/projected/16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4-kube-api-access-v4k8m\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr\" (UID: \"16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr" Nov 02 18:00:01.007467 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:01.007413 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9206222b-4b98-4c20-b48c-dc6d671948c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9\" (UID: \"9206222b-4b98-4c20-b48c-dc6d671948c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9" Nov 02 18:00:01.290330 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:01.275389 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr" Nov 02 18:00:01.290330 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:01.276052 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 02 18:00:01.305902 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:01.302215 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" podUID="b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.153:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:00:01.458900 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:01.409597 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" podUID="c3e6eb3c-0496-44c8-a45f-9b4c662756b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.155:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:00:01.458900 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:01.411809 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5"] Nov 02 18:00:01.465468 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:01.465418 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 02 18:00:01.504620 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:01.483076 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm" podUID="a4d08194-c737-4a26-9198-ecfd8e6554f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:00:01.504620 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:01.494261 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8svz9\" (UniqueName: \"kubernetes.io/projected/9206222b-4b98-4c20-b48c-dc6d671948c8-kube-api-access-8svz9\") pod \"maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9\" (UID: \"9206222b-4b98-4c20-b48c-dc6d671948c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9" Nov 02 18:00:01.523325 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:01.515364 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" podUID="198ca016-6343-4dda-999d-63a39d349f3a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:00:01.571610 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:01.571507 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 02 18:00:01.598297 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:01.582067 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 18:00:01.610327 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:01.599709 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:00:01.617697 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:01.617645 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": context deadline exceeded" Nov 02 18:00:01.683327 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:01.644602 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 02 18:00:01.724419 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:01.721723 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x"] Nov 02 18:00:01.738249 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:01.738218 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5" Nov 02 18:00:01.813337 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:01.800969 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9" Nov 02 18:00:01.827407 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:01.816657 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" podUID="33601c55-3e8f-41d4-a13e-1099a26334f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.151:8000/ready\": context deadline exceeded" Nov 02 18:00:01.864431 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:01.864161 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpbkx\" (UniqueName: \"kubernetes.io/projected/fab397a0-94f3-4ac9-9eea-36e8ca65ce1e-kube-api-access-fpbkx\") pod \"maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2\" (UID: \"fab397a0-94f3-4ac9-9eea-36e8ca65ce1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2" Nov 02 18:00:01.951816 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:01.951772 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5976425a-97ac-4edd-97d9-4720709d298b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5\" (UID: \"5976425a-97ac-4edd-97d9-4720709d298b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5" Nov 02 18:00:01.952113 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:01.952089 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkrx4\" (UniqueName: \"kubernetes.io/projected/5976425a-97ac-4edd-97d9-4720709d298b-kube-api-access-nkrx4\") pod \"maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5\" (UID: \"5976425a-97ac-4edd-97d9-4720709d298b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5" Nov 02 18:00:02.093403 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.069961 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkrx4\" (UniqueName: \"kubernetes.io/projected/5976425a-97ac-4edd-97d9-4720709d298b-kube-api-access-nkrx4\") pod \"maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5\" (UID: \"5976425a-97ac-4edd-97d9-4720709d298b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5" Nov 02 18:00:02.093403 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.070257 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5976425a-97ac-4edd-97d9-4720709d298b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5\" (UID: \"5976425a-97ac-4edd-97d9-4720709d298b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5" Nov 02 18:00:02.134343 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.112791 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5976425a-97ac-4edd-97d9-4720709d298b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5\" (UID: \"5976425a-97ac-4edd-97d9-4720709d298b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5" Nov 02 18:00:02.177328 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.151077 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2" Nov 02 18:00:02.223691 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.223661 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x" Nov 02 18:00:02.324249 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.223542 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv"] Nov 02 18:00:02.363115 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.263504 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" podUID="198ca016-6343-4dda-999d-63a39d349f3a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:00:02.363455 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.321582 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:00:02.364107 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.321765 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" podUID="b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.153:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:00:02.405401 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.405358 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhlr2\" (UniqueName: \"kubernetes.io/projected/8afe383f-c164-43f8-bf0e-992689371bde-kube-api-access-xhlr2\") pod \"maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x\" (UID: \"8afe383f-c164-43f8-bf0e-992689371bde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x" Nov 02 18:00:02.405966 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.405936 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8afe383f-c164-43f8-bf0e-992689371bde-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x\" (UID: \"8afe383f-c164-43f8-bf0e-992689371bde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x" Nov 02 18:00:02.489690 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.489644 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:00:02.502143 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.492700 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp"] Nov 02 18:00:02.539917 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.528837 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv" Nov 02 18:00:02.539917 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.537005 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efdb807d-4f96-44d5-b40a-61206ca67e7f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv\" (UID: \"efdb807d-4f96-44d5-b40a-61206ca67e7f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv" Nov 02 18:00:02.539917 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.537067 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhlr2\" (UniqueName: \"kubernetes.io/projected/8afe383f-c164-43f8-bf0e-992689371bde-kube-api-access-xhlr2\") pod \"maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x\" (UID: \"8afe383f-c164-43f8-bf0e-992689371bde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x" Nov 02 18:00:02.539917 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.537254 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8afe383f-c164-43f8-bf0e-992689371bde-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x\" (UID: \"8afe383f-c164-43f8-bf0e-992689371bde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x" Nov 02 18:00:02.539917 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.537398 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxkn2\" (UniqueName: \"kubernetes.io/projected/efdb807d-4f96-44d5-b40a-61206ca67e7f-kube-api-access-pxkn2\") pod \"maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv\" (UID: \"efdb807d-4f96-44d5-b40a-61206ca67e7f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv" Nov 02 18:00:02.539917 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.538563 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8afe383f-c164-43f8-bf0e-992689371bde-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x\" (UID: \"8afe383f-c164-43f8-bf0e-992689371bde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x" Nov 02 18:00:02.559951 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.544456 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp" Nov 02 18:00:02.573565 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.573389 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:00:02.607517 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.587399 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:00:02.607828 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.607788 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:00:02.668735 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.627424 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:00:02.668735 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.647513 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 02 18:00:02.668735 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.650714 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efdb807d-4f96-44d5-b40a-61206ca67e7f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv\" (UID: \"efdb807d-4f96-44d5-b40a-61206ca67e7f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv" Nov 02 18:00:02.668735 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.651132 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxkn2\" (UniqueName: \"kubernetes.io/projected/efdb807d-4f96-44d5-b40a-61206ca67e7f-kube-api-access-pxkn2\") pod \"maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv\" (UID: \"efdb807d-4f96-44d5-b40a-61206ca67e7f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv" Nov 02 18:00:02.668735 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.653145 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efdb807d-4f96-44d5-b40a-61206ca67e7f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv\" (UID: \"efdb807d-4f96-44d5-b40a-61206ca67e7f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv" Nov 02 18:00:02.675325 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.671713 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkrx4\" (UniqueName: \"kubernetes.io/projected/5976425a-97ac-4edd-97d9-4720709d298b-kube-api-access-nkrx4\") pod \"maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5\" (UID: \"5976425a-97ac-4edd-97d9-4720709d298b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5" Nov 02 18:00:02.702328 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.698192 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxkn2\" (UniqueName: \"kubernetes.io/projected/efdb807d-4f96-44d5-b40a-61206ca67e7f-kube-api-access-pxkn2\") pod \"maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv\" (UID: \"efdb807d-4f96-44d5-b40a-61206ca67e7f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv" Nov 02 18:00:02.767084 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.767045 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhlr2\" (UniqueName: \"kubernetes.io/projected/8afe383f-c164-43f8-bf0e-992689371bde-kube-api-access-xhlr2\") pod \"maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x\" (UID: \"8afe383f-c164-43f8-bf0e-992689371bde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x" Nov 02 18:00:02.790936 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.781795 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 18:00:02.790936 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.781832 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2"] Nov 02 18:00:02.790936 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.781889 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" Nov 02 18:00:02.790936 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.781908 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9"] Nov 02 18:00:02.816423 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.808496 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x" Nov 02 18:00:02.845763 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.818393 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eed364c-82b6-4fd8-8c04-4fef56994df6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp\" (UID: \"6eed364c-82b6-4fd8-8c04-4fef56994df6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp" Nov 02 18:00:02.845763 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.818729 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl7f9\" (UniqueName: \"kubernetes.io/projected/6eed364c-82b6-4fd8-8c04-4fef56994df6-kube-api-access-fl7f9\") pod \"maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp\" (UID: \"6eed364c-82b6-4fd8-8c04-4fef56994df6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp" Nov 02 18:00:02.845763 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.828651 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5" Nov 02 18:00:02.847179 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.847074 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" podUID="33601c55-3e8f-41d4-a13e-1099a26334f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.151:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:00:02.852594 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.851655 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv" Nov 02 18:00:02.998401 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.998294 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl7f9\" (UniqueName: \"kubernetes.io/projected/6eed364c-82b6-4fd8-8c04-4fef56994df6-kube-api-access-fl7f9\") pod \"maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp\" (UID: \"6eed364c-82b6-4fd8-8c04-4fef56994df6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp" Nov 02 18:00:02.999157 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:02.999128 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eed364c-82b6-4fd8-8c04-4fef56994df6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp\" (UID: \"6eed364c-82b6-4fd8-8c04-4fef56994df6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp" Nov 02 18:00:03.032915 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.032845 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eed364c-82b6-4fd8-8c04-4fef56994df6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp\" (UID: \"6eed364c-82b6-4fd8-8c04-4fef56994df6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp" Nov 02 18:00:03.057065 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.057023 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 18:00:03.058094 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.058068 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 18:00:03.058551 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.058535 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 18:00:03.059011 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.058995 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 18:00:03.081878 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.057907 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9" Nov 02 18:00:03.197612 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.082206 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 18:00:03.197919 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.197901 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 18:00:03.198109 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.198097 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" Nov 02 18:00:03.198278 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.198266 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 18:00:03.564516 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.560905 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9"] Nov 02 18:00:03.564516 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.560970 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5"] Nov 02 18:00:03.564516 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.560998 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x"] Nov 02 18:00:03.564516 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.561057 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 18:00:03.564516 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.157888 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" podUID="b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.153:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:00:03.564516 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.249063 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" podUID="c3e6eb3c-0496-44c8-a45f-9b4c662756b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.155:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:00:03.564516 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.382585 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl7f9\" (UniqueName: \"kubernetes.io/projected/6eed364c-82b6-4fd8-8c04-4fef56994df6-kube-api-access-fl7f9\") pod \"maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp\" (UID: \"6eed364c-82b6-4fd8-8c04-4fef56994df6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp" Nov 02 18:00:03.615277 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.578585 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" Nov 02 18:00:03.615277 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.582264 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 18:00:03.615277 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.586069 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 18:00:03.615277 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.586752 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" Nov 02 18:00:03.622109 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.622067 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:00:03.648013 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.647656 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp" Nov 02 18:00:03.649380 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.649328 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" podUID="198ca016-6343-4dda-999d-63a39d349f3a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:00:03.655487 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.655454 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e56641-f8f8-41be-bdcf-c31949402639-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9\" (UID: \"62e56641-f8f8-41be-bdcf-c31949402639\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9" Nov 02 18:00:03.663875 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.663844 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmfdj\" (UniqueName: \"kubernetes.io/projected/62e56641-f8f8-41be-bdcf-c31949402639-kube-api-access-jmfdj\") pod \"maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9\" (UID: \"62e56641-f8f8-41be-bdcf-c31949402639\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9" Nov 02 18:00:03.684810 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.673030 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 18:00:03.684810 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.673082 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 18:00:03.691238 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.691189 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 18:00:03.691408 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.691256 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 18:00:03.763432 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.763385 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp"] Nov 02 18:00:03.786255 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.785867 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e56641-f8f8-41be-bdcf-c31949402639-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9\" (UID: \"62e56641-f8f8-41be-bdcf-c31949402639\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9" Nov 02 18:00:03.787373 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.786605 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmfdj\" (UniqueName: \"kubernetes.io/projected/62e56641-f8f8-41be-bdcf-c31949402639-kube-api-access-jmfdj\") pod \"maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9\" (UID: \"62e56641-f8f8-41be-bdcf-c31949402639\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9" Nov 02 18:00:03.787373 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.787123 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e56641-f8f8-41be-bdcf-c31949402639-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9\" (UID: \"62e56641-f8f8-41be-bdcf-c31949402639\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9" Nov 02 18:00:03.859283 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.858521 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmfdj\" (UniqueName: \"kubernetes.io/projected/62e56641-f8f8-41be-bdcf-c31949402639-kube-api-access-jmfdj\") pod \"maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9\" (UID: \"62e56641-f8f8-41be-bdcf-c31949402639\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9" Nov 02 18:00:03.864993 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.864916 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv"] Nov 02 18:00:03.873260 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.872977 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9"] Nov 02 18:00:03.953470 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.953433 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9" Nov 02 18:00:03.960464 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:03.960432 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr"] Nov 02 18:00:04.139517 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:00:04.139482 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9206222b_4b98_4c20_b48c_dc6d671948c8.slice/crio-e9861d4cb142ec3a9f44ccf87b9d774d5a3e297a8afaf27dd69e5ebd742dbf68 WatchSource:0}: Error finding container e9861d4cb142ec3a9f44ccf87b9d774d5a3e297a8afaf27dd69e5ebd742dbf68: Status 404 returned error can't find the container with id e9861d4cb142ec3a9f44ccf87b9d774d5a3e297a8afaf27dd69e5ebd742dbf68 Nov 02 18:00:04.140203 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:04.140180 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp"] Nov 02 18:00:04.146583 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:04.146559 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9"] Nov 02 18:00:04.196819 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:00:04.196787 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefdb807d_4f96_44d5_b40a_61206ca67e7f.slice/crio-438d1d240943d271cbefa9282b2c7c1fcd3c3ff1030e5342049df9aa0e758101 WatchSource:0}: Error finding container 438d1d240943d271cbefa9282b2c7c1fcd3c3ff1030e5342049df9aa0e758101: Status 404 returned error can't find the container with id 438d1d240943d271cbefa9282b2c7c1fcd3c3ff1030e5342049df9aa0e758101 Nov 02 18:00:04.208228 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:04.208201 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv"] Nov 02 18:00:04.214358 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:04.214328 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5"] Nov 02 18:00:04.215894 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:00:04.215855 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5976425a_97ac_4edd_97d9_4720709d298b.slice/crio-cd267fe4b3201c8067c97e120200e46d9ec950c0a0996d792db07f94e70d8e38 WatchSource:0}: Error finding container cd267fe4b3201c8067c97e120200e46d9ec950c0a0996d792db07f94e70d8e38: Status 404 returned error can't find the container with id cd267fe4b3201c8067c97e120200e46d9ec950c0a0996d792db07f94e70d8e38 Nov 02 18:00:04.242066 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:04.241925 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv" event={"ID":"efdb807d-4f96-44d5-b40a-61206ca67e7f","Type":"ContainerStarted","Data":"438d1d240943d271cbefa9282b2c7c1fcd3c3ff1030e5342049df9aa0e758101"} Nov 02 18:00:04.251468 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:04.251437 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9" event={"ID":"9206222b-4b98-4c20-b48c-dc6d671948c8","Type":"ContainerStarted","Data":"e9861d4cb142ec3a9f44ccf87b9d774d5a3e297a8afaf27dd69e5ebd742dbf68"} Nov 02 18:00:04.254361 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:04.254333 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x"] Nov 02 18:00:04.255056 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:04.255018 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5" event={"ID":"5976425a-97ac-4edd-97d9-4720709d298b","Type":"ContainerStarted","Data":"cd267fe4b3201c8067c97e120200e46d9ec950c0a0996d792db07f94e70d8e38"} Nov 02 18:00:04.257344 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:04.257287 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr" event={"ID":"16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4","Type":"ContainerStarted","Data":"3e3163c8e3c6e7325669a49d478b277d543be4dbcd20cdf6d9a146e8fa4d42fb"} Nov 02 18:00:04.259230 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:04.259209 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp" event={"ID":"6eed364c-82b6-4fd8-8c04-4fef56994df6","Type":"ContainerStarted","Data":"4fe5bc6d9b57d6a7a5992b07a007e07630690fc049082016cbfa6db4b19111d0"} Nov 02 18:00:04.287052 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:04.283516 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9"] Nov 02 18:00:04.287240 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:00:04.287195 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e56641_f8f8_41be_bdcf_c31949402639.slice/crio-a492921ebbdad1a8cabbde5c6d0a08cada87e0955d46846fde855de82aa1e463 WatchSource:0}: Error finding container a492921ebbdad1a8cabbde5c6d0a08cada87e0955d46846fde855de82aa1e463: Status 404 returned error can't find the container with id a492921ebbdad1a8cabbde5c6d0a08cada87e0955d46846fde855de82aa1e463 Nov 02 18:00:04.318077 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:04.317903 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2"] Nov 02 18:00:05.294048 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:05.293923 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9" event={"ID":"62e56641-f8f8-41be-bdcf-c31949402639","Type":"ContainerStarted","Data":"a492921ebbdad1a8cabbde5c6d0a08cada87e0955d46846fde855de82aa1e463"} Nov 02 18:00:05.298970 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:05.298847 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x" event={"ID":"8afe383f-c164-43f8-bf0e-992689371bde","Type":"ContainerStarted","Data":"4d8cb6742e4f6e32ffbe7e09d85306b811da153b5654e4ffa79431a834d41d35"} Nov 02 18:00:05.302956 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:05.302925 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2" event={"ID":"fab397a0-94f3-4ac9-9eea-36e8ca65ce1e","Type":"ContainerStarted","Data":"6ca03e144a89722da88069e42b1ce371dde83a271524c0c47b534dccde67cc24"} Nov 02 18:00:07.385258 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:07.385222 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg"] Nov 02 18:00:07.409552 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:07.409520 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:00:07.414172 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:07.414137 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg"] Nov 02 18:00:07.527394 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:07.527349 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istio-token\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:00:07.527602 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:07.527518 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istio-envoy\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:00:07.527602 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:07.527582 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-credential-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:00:07.527743 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:07.527655 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istio-data\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:00:07.527743 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:07.527695 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istio-podinfo\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:00:07.527843 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:07.527826 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l458b\" (UniqueName: \"kubernetes.io/projected/e304caf1-38c3-4cbb-9c2f-db257e76ada7-kube-api-access-l458b\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:00:07.527981 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:07.527961 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-workload-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:00:07.528101 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:07.527994 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istiod-ca-cert\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:00:07.528154 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:07.528111 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-workload-certs\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:00:07.630356 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:07.630280 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-workload-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:00:07.630549 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:07.630488 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istiod-ca-cert\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:00:07.630713 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:07.630694 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-workload-certs\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:00:07.630804 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:07.630732 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-workload-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:00:07.630898 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:07.630878 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istio-token\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:00:07.631065 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:07.631046 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istio-envoy\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:00:07.631128 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:07.631116 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-credential-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:00:07.631192 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:07.631180 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istio-data\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:00:07.631253 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:07.631219 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istio-podinfo\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:00:07.631390 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:07.631366 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istiod-ca-cert\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:00:07.631455 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:07.631388 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l458b\" (UniqueName: \"kubernetes.io/projected/e304caf1-38c3-4cbb-9c2f-db257e76ada7-kube-api-access-l458b\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:00:07.631914 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:07.631888 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-workload-certs\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:00:07.632861 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:07.632839 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-credential-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:00:07.633113 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:07.633091 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istio-data\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:00:07.643733 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:07.643653 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istio-podinfo\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:00:07.652810 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:07.652780 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istio-envoy\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:00:07.656780 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:07.656688 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l458b\" (UniqueName: \"kubernetes.io/projected/e304caf1-38c3-4cbb-9c2f-db257e76ada7-kube-api-access-l458b\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:00:07.656780 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:07.656731 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istio-token\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:00:07.746030 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:07.745996 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:00:08.138593 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:00:08.138457 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode304caf1_38c3_4cbb_9c2f_db257e76ada7.slice/crio-aed1fea339a4ca98e72f7297df6396a73e62ceaa5b6da9a0b8a4aff75e21b2ef WatchSource:0}: Error finding container aed1fea339a4ca98e72f7297df6396a73e62ceaa5b6da9a0b8a4aff75e21b2ef: Status 404 returned error can't find the container with id aed1fea339a4ca98e72f7297df6396a73e62ceaa5b6da9a0b8a4aff75e21b2ef Nov 02 18:00:08.142570 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:08.142551 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg"] Nov 02 18:00:08.143171 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:08.143130 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:00:08.143277 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:08.143211 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:00:08.143277 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:08.143249 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:00:08.424680 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:08.423345 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" event={"ID":"e304caf1-38c3-4cbb-9c2f-db257e76ada7","Type":"ContainerStarted","Data":"aed1fea339a4ca98e72f7297df6396a73e62ceaa5b6da9a0b8a4aff75e21b2ef"} Nov 02 18:00:08.747383 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:08.747271 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:00:09.232333 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:00:09.229853 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:00:09.442701 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:09.442662 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" event={"ID":"e304caf1-38c3-4cbb-9c2f-db257e76ada7","Type":"ContainerStarted","Data":"08ad3857b2a6d05b15a41996225a0a4807f64a50355e002fff80318bcf031a0f"} Nov 02 18:00:09.749767 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:09.749406 4471 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 02 18:00:09.749767 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:09.749469 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" podUID="e304caf1-38c3-4cbb-9c2f-db257e76ada7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 02 18:00:10.487827 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:10.487766 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x" event={"ID":"8afe383f-c164-43f8-bf0e-992689371bde","Type":"ContainerStarted","Data":"58bd6e17c36423b51d12d40fb37b19236a4c430d8e293fb0d73e4cc43aed51de"} Nov 02 18:00:10.490771 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:10.490742 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp" event={"ID":"6eed364c-82b6-4fd8-8c04-4fef56994df6","Type":"ContainerStarted","Data":"01a692641a261b35ca9bdf1fb166d25baf933324438cfeeb838d577f34fb0986"} Nov 02 18:00:10.552063 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:10.551998 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x" podStartSLOduration=5.549981288 podStartE2EDuration="10.551978597s" totalImagesPullingTime="5.001997309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:10.547744204 +0000 UTC m=+1828.243099696" watchObservedRunningTime="2026-11-02 18:00:10.551978597 +0000 UTC m=+1828.247334073" Nov 02 18:00:10.553533 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:10.553484 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" podStartSLOduration=3.553468164 podStartE2EDuration="3.553468164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:08.512107116 +0000 UTC m=+1826.207462615" watchObservedRunningTime="2026-11-02 18:00:10.553468164 +0000 UTC m=+1828.248823642" Nov 02 18:00:10.621872 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:10.621811 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp" podStartSLOduration=5.22509348 podStartE2EDuration="10.621789436s" totalImagesPullingTime="5.396695956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:10.615229467 +0000 UTC m=+1828.310584940" watchObservedRunningTime="2026-11-02 18:00:10.621789436 +0000 UTC m=+1828.317144912" Nov 02 18:00:10.747201 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:10.747168 4471 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 02 18:00:10.747386 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:10.747233 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" podUID="e304caf1-38c3-4cbb-9c2f-db257e76ada7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 02 18:00:11.501837 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:11.501741 4471 generic.go:356] "Generic (PLEG): container finished" podID="6eed364c-82b6-4fd8-8c04-4fef56994df6" containerID="01a692641a261b35ca9bdf1fb166d25baf933324438cfeeb838d577f34fb0986" exitCode=60 Nov 02 18:00:11.502259 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:11.501827 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp" event={"ID":"6eed364c-82b6-4fd8-8c04-4fef56994df6","Type":"ContainerDied","Data":"01a692641a261b35ca9bdf1fb166d25baf933324438cfeeb838d577f34fb0986"} Nov 02 18:00:11.502259 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:11.502219 4471 scope.go:122] "RemoveContainer" containerID="01a692641a261b35ca9bdf1fb166d25baf933324438cfeeb838d577f34fb0986" Nov 02 18:00:11.504154 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:11.504034 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2" event={"ID":"fab397a0-94f3-4ac9-9eea-36e8ca65ce1e","Type":"ContainerStarted","Data":"ecd697a2e04735531fa5fd29c9f8c5f2c34a76ed79e89cb0387712fb51fb355b"} Nov 02 18:00:11.506031 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:11.506008 4471 generic.go:356] "Generic (PLEG): container finished" podID="efdb807d-4f96-44d5-b40a-61206ca67e7f" containerID="62478f29ce57c26edff0e75a611d73c614ea4e3ea6a9c97e2953a48a2f106dcf" exitCode=60 Nov 02 18:00:11.506188 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:11.506166 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv" event={"ID":"efdb807d-4f96-44d5-b40a-61206ca67e7f","Type":"ContainerDied","Data":"62478f29ce57c26edff0e75a611d73c614ea4e3ea6a9c97e2953a48a2f106dcf"} Nov 02 18:00:11.506430 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:11.506411 4471 scope.go:122] "RemoveContainer" containerID="62478f29ce57c26edff0e75a611d73c614ea4e3ea6a9c97e2953a48a2f106dcf" Nov 02 18:00:11.508899 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:11.508798 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9" event={"ID":"9206222b-4b98-4c20-b48c-dc6d671948c8","Type":"ContainerStarted","Data":"d1f6254ed15d520c6d432c744b22aa22892486bc8f585cade246f7515ca18172"} Nov 02 18:00:11.510592 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:11.510544 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5" event={"ID":"5976425a-97ac-4edd-97d9-4720709d298b","Type":"ContainerStarted","Data":"c0bc2fd853f94831e206cf1c1830df93a41f5ec303299b18776f3bdf0e0e3c12"} Nov 02 18:00:11.512453 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:11.512430 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9" event={"ID":"62e56641-f8f8-41be-bdcf-c31949402639","Type":"ContainerStarted","Data":"cad30762d170715087e6d0b98f785df8d97e5cf7ee21f7b75293ccfd50de6894"} Nov 02 18:00:11.514227 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:11.514198 4471 generic.go:356] "Generic (PLEG): container finished" podID="8afe383f-c164-43f8-bf0e-992689371bde" containerID="58bd6e17c36423b51d12d40fb37b19236a4c430d8e293fb0d73e4cc43aed51de" exitCode=60 Nov 02 18:00:11.514370 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:11.514231 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x" event={"ID":"8afe383f-c164-43f8-bf0e-992689371bde","Type":"ContainerDied","Data":"58bd6e17c36423b51d12d40fb37b19236a4c430d8e293fb0d73e4cc43aed51de"} Nov 02 18:00:11.514561 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:11.514541 4471 scope.go:122] "RemoveContainer" containerID="58bd6e17c36423b51d12d40fb37b19236a4c430d8e293fb0d73e4cc43aed51de" Nov 02 18:00:11.590438 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:11.590366 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5" podStartSLOduration=5.313052577 podStartE2EDuration="11.590348462s" totalImagesPullingTime="6.277295885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:11.589450699 +0000 UTC m=+1829.284806200" watchObservedRunningTime="2026-11-02 18:00:11.590348462 +0000 UTC m=+1829.285703961" Nov 02 18:00:11.631738 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:11.631690 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9" podStartSLOduration=5.428587905 podStartE2EDuration="11.631676449s" totalImagesPullingTime="6.203088544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:11.630622708 +0000 UTC m=+1829.325978182" watchObservedRunningTime="2026-11-02 18:00:11.631676449 +0000 UTC m=+1829.327031915" Nov 02 18:00:11.694299 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:11.694240 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2" podStartSLOduration=5.514932319 podStartE2EDuration="11.69421867s" totalImagesPullingTime="6.179286351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:11.6792634 +0000 UTC m=+1829.374618876" watchObservedRunningTime="2026-11-02 18:00:11.69421867 +0000 UTC m=+1829.389574147" Nov 02 18:00:11.726899 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:11.726849 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9" podStartSLOduration=5.363357348 podStartE2EDuration="11.726834603s" totalImagesPullingTime="6.363477255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:11.725760175 +0000 UTC m=+1829.421115648" watchObservedRunningTime="2026-11-02 18:00:11.726834603 +0000 UTC m=+1829.422190076" Nov 02 18:00:11.747247 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:11.747209 4471 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 02 18:00:11.747396 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:11.747267 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" podUID="e304caf1-38c3-4cbb-9c2f-db257e76ada7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 02 18:00:12.524495 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:12.524462 4471 generic.go:356] "Generic (PLEG): container finished" podID="9206222b-4b98-4c20-b48c-dc6d671948c8" containerID="d1f6254ed15d520c6d432c744b22aa22892486bc8f585cade246f7515ca18172" exitCode=60 Nov 02 18:00:12.524953 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:12.524548 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9" event={"ID":"9206222b-4b98-4c20-b48c-dc6d671948c8","Type":"ContainerDied","Data":"d1f6254ed15d520c6d432c744b22aa22892486bc8f585cade246f7515ca18172"} Nov 02 18:00:12.526833 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:12.526558 4471 scope.go:122] "RemoveContainer" containerID="d1f6254ed15d520c6d432c744b22aa22892486bc8f585cade246f7515ca18172" Nov 02 18:00:12.526833 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:12.526595 4471 generic.go:356] "Generic (PLEG): container finished" podID="5976425a-97ac-4edd-97d9-4720709d298b" containerID="c0bc2fd853f94831e206cf1c1830df93a41f5ec303299b18776f3bdf0e0e3c12" exitCode=60 Nov 02 18:00:12.526833 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:12.526649 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5" event={"ID":"5976425a-97ac-4edd-97d9-4720709d298b","Type":"ContainerDied","Data":"c0bc2fd853f94831e206cf1c1830df93a41f5ec303299b18776f3bdf0e0e3c12"} Nov 02 18:00:12.527238 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:12.527087 4471 scope.go:122] "RemoveContainer" containerID="c0bc2fd853f94831e206cf1c1830df93a41f5ec303299b18776f3bdf0e0e3c12" Nov 02 18:00:12.529263 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:12.529194 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr" event={"ID":"16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4","Type":"ContainerStarted","Data":"a1c046faf0de134bce13dfc39f410d94441e777f2f2ea8f93597e2bc25c89f02"} Nov 02 18:00:12.532793 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:12.532773 4471 generic.go:356] "Generic (PLEG): container finished" podID="62e56641-f8f8-41be-bdcf-c31949402639" containerID="cad30762d170715087e6d0b98f785df8d97e5cf7ee21f7b75293ccfd50de6894" exitCode=60 Nov 02 18:00:12.532914 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:12.532880 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9" event={"ID":"62e56641-f8f8-41be-bdcf-c31949402639","Type":"ContainerDied","Data":"cad30762d170715087e6d0b98f785df8d97e5cf7ee21f7b75293ccfd50de6894"} Nov 02 18:00:12.533181 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:12.533161 4471 scope.go:122] "RemoveContainer" containerID="cad30762d170715087e6d0b98f785df8d97e5cf7ee21f7b75293ccfd50de6894" Nov 02 18:00:12.551506 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:12.551455 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x" event={"ID":"8afe383f-c164-43f8-bf0e-992689371bde","Type":"ContainerStarted","Data":"7064f566b412caa86d08b6d717b6ef5798dac3229bfc4597aadba758c017f241"} Nov 02 18:00:12.554137 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:12.554105 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp" event={"ID":"6eed364c-82b6-4fd8-8c04-4fef56994df6","Type":"ContainerStarted","Data":"61e83b7a81a510e2c854c957db784af701698689f7864c50b5645a9fa4d7b6c2"} Nov 02 18:00:12.590879 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:12.590839 4471 generic.go:356] "Generic (PLEG): container finished" podID="fab397a0-94f3-4ac9-9eea-36e8ca65ce1e" containerID="ecd697a2e04735531fa5fd29c9f8c5f2c34a76ed79e89cb0387712fb51fb355b" exitCode=60 Nov 02 18:00:12.591048 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:12.590964 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2" event={"ID":"fab397a0-94f3-4ac9-9eea-36e8ca65ce1e","Type":"ContainerDied","Data":"ecd697a2e04735531fa5fd29c9f8c5f2c34a76ed79e89cb0387712fb51fb355b"} Nov 02 18:00:12.591540 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:12.591514 4471 scope.go:122] "RemoveContainer" containerID="ecd697a2e04735531fa5fd29c9f8c5f2c34a76ed79e89cb0387712fb51fb355b" Nov 02 18:00:12.604581 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:12.602795 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv" event={"ID":"efdb807d-4f96-44d5-b40a-61206ca67e7f","Type":"ContainerStarted","Data":"504c926100fc189ce4e98a99d0dec97faaf3388addcf2a3e6ddedce97c6b626f"} Nov 02 18:00:12.742285 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:12.742237 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr" podStartSLOduration=5.329602457 podStartE2EDuration="12.742223835s" totalImagesPullingTime="7.412621378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:12.694697496 +0000 UTC m=+1830.390052969" watchObservedRunningTime="2026-11-02 18:00:12.742223835 +0000 UTC m=+1830.437579308" Nov 02 18:00:12.746894 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:12.746866 4471 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 02 18:00:12.747414 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:12.747382 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" podUID="e304caf1-38c3-4cbb-9c2f-db257e76ada7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 02 18:00:12.918394 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:12.918341 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv" podStartSLOduration=7.204367329 podStartE2EDuration="12.918325908s" totalImagesPullingTime="5.713958579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:12.915657138 +0000 UTC m=+1830.611012630" watchObservedRunningTime="2026-11-02 18:00:12.918325908 +0000 UTC m=+1830.613681378" Nov 02 18:00:13.614288 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:13.614249 4471 generic.go:356] "Generic (PLEG): container finished" podID="efdb807d-4f96-44d5-b40a-61206ca67e7f" containerID="504c926100fc189ce4e98a99d0dec97faaf3388addcf2a3e6ddedce97c6b626f" exitCode=60 Nov 02 18:00:13.614726 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:13.614451 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv" event={"ID":"efdb807d-4f96-44d5-b40a-61206ca67e7f","Type":"ContainerDied","Data":"504c926100fc189ce4e98a99d0dec97faaf3388addcf2a3e6ddedce97c6b626f"} Nov 02 18:00:13.614726 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:13.614514 4471 scope.go:122] "RemoveContainer" containerID="62478f29ce57c26edff0e75a611d73c614ea4e3ea6a9c97e2953a48a2f106dcf" Nov 02 18:00:13.614818 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:13.614731 4471 scope.go:122] "RemoveContainer" containerID="504c926100fc189ce4e98a99d0dec97faaf3388addcf2a3e6ddedce97c6b626f" Nov 02 18:00:13.615615 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:00:13.615273 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv_odh-ai-gateway-infra(efdb807d-4f96-44d5-b40a-61206ca67e7f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv" podUID="efdb807d-4f96-44d5-b40a-61206ca67e7f" Nov 02 18:00:13.616835 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:13.616458 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9" event={"ID":"9206222b-4b98-4c20-b48c-dc6d671948c8","Type":"ContainerStarted","Data":"fb05659122969b033d2a9ce1f396e3823561ebf004826a3e0dba62c1d3de79e8"} Nov 02 18:00:13.618947 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:13.618929 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5" event={"ID":"5976425a-97ac-4edd-97d9-4720709d298b","Type":"ContainerStarted","Data":"19462df9b74be4d63d3e8132c8a4631e8e248e9543aba0d7dea6afab3be67325"} Nov 02 18:00:13.620572 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:13.620540 4471 generic.go:356] "Generic (PLEG): container finished" podID="16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4" containerID="a1c046faf0de134bce13dfc39f410d94441e777f2f2ea8f93597e2bc25c89f02" exitCode=60 Nov 02 18:00:13.620651 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:13.620600 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr" event={"ID":"16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4","Type":"ContainerDied","Data":"a1c046faf0de134bce13dfc39f410d94441e777f2f2ea8f93597e2bc25c89f02"} Nov 02 18:00:13.620829 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:13.620811 4471 scope.go:122] "RemoveContainer" containerID="a1c046faf0de134bce13dfc39f410d94441e777f2f2ea8f93597e2bc25c89f02" Nov 02 18:00:13.622800 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:13.622779 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9" event={"ID":"62e56641-f8f8-41be-bdcf-c31949402639","Type":"ContainerStarted","Data":"b00d34d057953fe74795355fb34ef107bfa6eb88173a60fea0015c6afcd0c7de"} Nov 02 18:00:13.624804 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:13.624781 4471 generic.go:356] "Generic (PLEG): container finished" podID="8afe383f-c164-43f8-bf0e-992689371bde" containerID="7064f566b412caa86d08b6d717b6ef5798dac3229bfc4597aadba758c017f241" exitCode=60 Nov 02 18:00:13.625084 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:13.625069 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x" event={"ID":"8afe383f-c164-43f8-bf0e-992689371bde","Type":"ContainerDied","Data":"7064f566b412caa86d08b6d717b6ef5798dac3229bfc4597aadba758c017f241"} Nov 02 18:00:13.625494 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:13.625480 4471 scope.go:122] "RemoveContainer" containerID="7064f566b412caa86d08b6d717b6ef5798dac3229bfc4597aadba758c017f241" Nov 02 18:00:13.626472 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:00:13.626450 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x_odh-ai-gateway-infra(8afe383f-c164-43f8-bf0e-992689371bde)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x" podUID="8afe383f-c164-43f8-bf0e-992689371bde" Nov 02 18:00:13.627769 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:13.627737 4471 generic.go:356] "Generic (PLEG): container finished" podID="6eed364c-82b6-4fd8-8c04-4fef56994df6" containerID="61e83b7a81a510e2c854c957db784af701698689f7864c50b5645a9fa4d7b6c2" exitCode=60 Nov 02 18:00:13.627864 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:13.627772 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp" event={"ID":"6eed364c-82b6-4fd8-8c04-4fef56994df6","Type":"ContainerDied","Data":"61e83b7a81a510e2c854c957db784af701698689f7864c50b5645a9fa4d7b6c2"} Nov 02 18:00:13.628047 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:13.628030 4471 scope.go:122] "RemoveContainer" containerID="61e83b7a81a510e2c854c957db784af701698689f7864c50b5645a9fa4d7b6c2" Nov 02 18:00:13.628403 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:00:13.628379 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp_odh-ai-gateway-infra(6eed364c-82b6-4fd8-8c04-4fef56994df6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp" podUID="6eed364c-82b6-4fd8-8c04-4fef56994df6" Nov 02 18:00:13.630247 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:13.630226 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2" event={"ID":"fab397a0-94f3-4ac9-9eea-36e8ca65ce1e","Type":"ContainerStarted","Data":"b927bb622aae8eccceb2b9ef032369ef04160a263a270759519b1d86f6eb0073"} Nov 02 18:00:13.637226 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:13.637195 4471 scope.go:122] "RemoveContainer" containerID="58bd6e17c36423b51d12d40fb37b19236a4c430d8e293fb0d73e4cc43aed51de" Nov 02 18:00:13.697975 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:13.697952 4471 scope.go:122] "RemoveContainer" containerID="01a692641a261b35ca9bdf1fb166d25baf933324438cfeeb838d577f34fb0986" Nov 02 18:00:13.747769 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:13.747736 4471 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 02 18:00:13.747901 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:13.747792 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" podUID="e304caf1-38c3-4cbb-9c2f-db257e76ada7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 02 18:00:14.645966 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:14.645934 4471 scope.go:122] "RemoveContainer" containerID="504c926100fc189ce4e98a99d0dec97faaf3388addcf2a3e6ddedce97c6b626f" Nov 02 18:00:14.646719 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:00:14.646464 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv_odh-ai-gateway-infra(efdb807d-4f96-44d5-b40a-61206ca67e7f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv" podUID="efdb807d-4f96-44d5-b40a-61206ca67e7f" Nov 02 18:00:14.647970 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:14.647944 4471 generic.go:356] "Generic (PLEG): container finished" podID="9206222b-4b98-4c20-b48c-dc6d671948c8" containerID="fb05659122969b033d2a9ce1f396e3823561ebf004826a3e0dba62c1d3de79e8" exitCode=60 Nov 02 18:00:14.648157 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:14.648006 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9" event={"ID":"9206222b-4b98-4c20-b48c-dc6d671948c8","Type":"ContainerDied","Data":"fb05659122969b033d2a9ce1f396e3823561ebf004826a3e0dba62c1d3de79e8"} Nov 02 18:00:14.648222 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:14.648196 4471 scope.go:122] "RemoveContainer" containerID="d1f6254ed15d520c6d432c744b22aa22892486bc8f585cade246f7515ca18172" Nov 02 18:00:14.648373 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:14.648355 4471 scope.go:122] "RemoveContainer" containerID="fb05659122969b033d2a9ce1f396e3823561ebf004826a3e0dba62c1d3de79e8" Nov 02 18:00:14.649350 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:00:14.648802 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9_odh-ai-gateway-infra(9206222b-4b98-4c20-b48c-dc6d671948c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9" podUID="9206222b-4b98-4c20-b48c-dc6d671948c8" Nov 02 18:00:14.651218 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:14.651194 4471 generic.go:356] "Generic (PLEG): container finished" podID="5976425a-97ac-4edd-97d9-4720709d298b" containerID="19462df9b74be4d63d3e8132c8a4631e8e248e9543aba0d7dea6afab3be67325" exitCode=60 Nov 02 18:00:14.651299 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:14.651245 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5" event={"ID":"5976425a-97ac-4edd-97d9-4720709d298b","Type":"ContainerDied","Data":"19462df9b74be4d63d3e8132c8a4631e8e248e9543aba0d7dea6afab3be67325"} Nov 02 18:00:14.651641 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:14.651624 4471 scope.go:122] "RemoveContainer" containerID="19462df9b74be4d63d3e8132c8a4631e8e248e9543aba0d7dea6afab3be67325" Nov 02 18:00:14.651939 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:00:14.651919 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5_odh-ai-gateway-infra(5976425a-97ac-4edd-97d9-4720709d298b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5" podUID="5976425a-97ac-4edd-97d9-4720709d298b" Nov 02 18:00:14.653600 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:14.653574 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr" event={"ID":"16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4","Type":"ContainerStarted","Data":"c1920c940c2c71739f33b54c1dd568153d8707c714975dd99e8212ae2e8216d2"} Nov 02 18:00:14.662828 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:14.662802 4471 generic.go:356] "Generic (PLEG): container finished" podID="62e56641-f8f8-41be-bdcf-c31949402639" containerID="b00d34d057953fe74795355fb34ef107bfa6eb88173a60fea0015c6afcd0c7de" exitCode=60 Nov 02 18:00:14.662948 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:14.662932 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9" event={"ID":"62e56641-f8f8-41be-bdcf-c31949402639","Type":"ContainerDied","Data":"b00d34d057953fe74795355fb34ef107bfa6eb88173a60fea0015c6afcd0c7de"} Nov 02 18:00:14.663645 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:14.663379 4471 scope.go:122] "RemoveContainer" containerID="b00d34d057953fe74795355fb34ef107bfa6eb88173a60fea0015c6afcd0c7de" Nov 02 18:00:14.663765 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:00:14.663744 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9_odh-ai-gateway-infra(62e56641-f8f8-41be-bdcf-c31949402639)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9" podUID="62e56641-f8f8-41be-bdcf-c31949402639" Nov 02 18:00:14.665538 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:14.665381 4471 scope.go:122] "RemoveContainer" containerID="7064f566b412caa86d08b6d717b6ef5798dac3229bfc4597aadba758c017f241" Nov 02 18:00:14.665754 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:00:14.665729 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x_odh-ai-gateway-infra(8afe383f-c164-43f8-bf0e-992689371bde)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x" podUID="8afe383f-c164-43f8-bf0e-992689371bde" Nov 02 18:00:14.667463 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:14.667445 4471 scope.go:122] "RemoveContainer" containerID="61e83b7a81a510e2c854c957db784af701698689f7864c50b5645a9fa4d7b6c2" Nov 02 18:00:14.667754 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:00:14.667727 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp_odh-ai-gateway-infra(6eed364c-82b6-4fd8-8c04-4fef56994df6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp" podUID="6eed364c-82b6-4fd8-8c04-4fef56994df6" Nov 02 18:00:14.669793 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:14.669765 4471 generic.go:356] "Generic (PLEG): container finished" podID="fab397a0-94f3-4ac9-9eea-36e8ca65ce1e" containerID="b927bb622aae8eccceb2b9ef032369ef04160a263a270759519b1d86f6eb0073" exitCode=60 Nov 02 18:00:14.669911 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:14.669811 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2" event={"ID":"fab397a0-94f3-4ac9-9eea-36e8ca65ce1e","Type":"ContainerDied","Data":"b927bb622aae8eccceb2b9ef032369ef04160a263a270759519b1d86f6eb0073"} Nov 02 18:00:14.670130 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:14.670109 4471 scope.go:122] "RemoveContainer" containerID="b927bb622aae8eccceb2b9ef032369ef04160a263a270759519b1d86f6eb0073" Nov 02 18:00:14.670489 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:00:14.670465 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2_odh-ai-gateway-infra(fab397a0-94f3-4ac9-9eea-36e8ca65ce1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2" podUID="fab397a0-94f3-4ac9-9eea-36e8ca65ce1e" Nov 02 18:00:14.671673 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:14.671655 4471 scope.go:122] "RemoveContainer" containerID="c0bc2fd853f94831e206cf1c1830df93a41f5ec303299b18776f3bdf0e0e3c12" Nov 02 18:00:14.691128 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:14.691106 4471 scope.go:122] "RemoveContainer" containerID="cad30762d170715087e6d0b98f785df8d97e5cf7ee21f7b75293ccfd50de6894" Nov 02 18:00:14.712381 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:14.712357 4471 scope.go:122] "RemoveContainer" containerID="ecd697a2e04735531fa5fd29c9f8c5f2c34a76ed79e89cb0387712fb51fb355b" Nov 02 18:00:14.747661 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:14.747628 4471 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 02 18:00:14.747776 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:14.747687 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" podUID="e304caf1-38c3-4cbb-9c2f-db257e76ada7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 02 18:00:15.681702 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:15.681673 4471 scope.go:122] "RemoveContainer" containerID="b927bb622aae8eccceb2b9ef032369ef04160a263a270759519b1d86f6eb0073" Nov 02 18:00:15.682199 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:00:15.682169 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2_odh-ai-gateway-infra(fab397a0-94f3-4ac9-9eea-36e8ca65ce1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2" podUID="fab397a0-94f3-4ac9-9eea-36e8ca65ce1e" Nov 02 18:00:15.683621 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:15.683594 4471 scope.go:122] "RemoveContainer" containerID="fb05659122969b033d2a9ce1f396e3823561ebf004826a3e0dba62c1d3de79e8" Nov 02 18:00:15.684021 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:00:15.683997 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9_odh-ai-gateway-infra(9206222b-4b98-4c20-b48c-dc6d671948c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9" podUID="9206222b-4b98-4c20-b48c-dc6d671948c8" Nov 02 18:00:15.685325 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:15.685295 4471 scope.go:122] "RemoveContainer" containerID="19462df9b74be4d63d3e8132c8a4631e8e248e9543aba0d7dea6afab3be67325" Nov 02 18:00:15.685692 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:00:15.685668 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5_odh-ai-gateway-infra(5976425a-97ac-4edd-97d9-4720709d298b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5" podUID="5976425a-97ac-4edd-97d9-4720709d298b" Nov 02 18:00:15.686953 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:15.686931 4471 generic.go:356] "Generic (PLEG): container finished" podID="16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4" containerID="c1920c940c2c71739f33b54c1dd568153d8707c714975dd99e8212ae2e8216d2" exitCode=60 Nov 02 18:00:15.687024 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:15.686991 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr" event={"ID":"16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4","Type":"ContainerDied","Data":"c1920c940c2c71739f33b54c1dd568153d8707c714975dd99e8212ae2e8216d2"} Nov 02 18:00:15.687071 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:15.687048 4471 scope.go:122] "RemoveContainer" containerID="a1c046faf0de134bce13dfc39f410d94441e777f2f2ea8f93597e2bc25c89f02" Nov 02 18:00:15.687315 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:15.687282 4471 scope.go:122] "RemoveContainer" containerID="c1920c940c2c71739f33b54c1dd568153d8707c714975dd99e8212ae2e8216d2" Nov 02 18:00:15.687722 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:00:15.687691 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr_odh-ai-gateway-infra(16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr" podUID="16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4" Nov 02 18:00:15.689112 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:15.689097 4471 scope.go:122] "RemoveContainer" containerID="b00d34d057953fe74795355fb34ef107bfa6eb88173a60fea0015c6afcd0c7de" Nov 02 18:00:15.689412 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:00:15.689393 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9_odh-ai-gateway-infra(62e56641-f8f8-41be-bdcf-c31949402639)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9" podUID="62e56641-f8f8-41be-bdcf-c31949402639" Nov 02 18:00:15.796354 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:15.793610 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:00:15.796354 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:15.794951 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:00:15.796354 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:15.796008 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:00:16.701668 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:16.701639 4471 scope.go:122] "RemoveContainer" containerID="c1920c940c2c71739f33b54c1dd568153d8707c714975dd99e8212ae2e8216d2" Nov 02 18:00:16.702098 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:00:16.702071 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr_odh-ai-gateway-infra(16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr" podUID="16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4" Nov 02 18:00:20.583856 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:20.583815 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl"] Nov 02 18:00:20.623525 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:20.623492 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:00:20.629458 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:20.629415 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl"] Nov 02 18:00:20.693855 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:20.693809 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-istio-envoy\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-zw5jl\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:00:20.694047 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:20.693953 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6183d0fa-f089-47f0-97a0-163160fc9f25-istio-podinfo\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-zw5jl\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:00:20.694119 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:20.694044 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6183d0fa-f089-47f0-97a0-163160fc9f25-istiod-ca-cert\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-zw5jl\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:00:20.694189 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:20.694171 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-workload-socket\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-zw5jl\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:00:20.694248 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:20.694229 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-credential-socket\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-zw5jl\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:00:20.694375 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:20.694278 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvpw2\" (UniqueName: \"kubernetes.io/projected/6183d0fa-f089-47f0-97a0-163160fc9f25-kube-api-access-pvpw2\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-zw5jl\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:00:20.694375 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:20.694350 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-workload-certs\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-zw5jl\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:00:20.694496 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:20.694458 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-istio-data\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-zw5jl\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:00:20.694496 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:20.694483 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6183d0fa-f089-47f0-97a0-163160fc9f25-istio-token\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-zw5jl\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:00:20.724753 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:20.724716 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-872354-openshift-default-dockercfg-rdxd7\"" Nov 02 18:00:20.796488 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:20.796452 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-istio-envoy\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-zw5jl\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:00:20.796670 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:20.796586 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6183d0fa-f089-47f0-97a0-163160fc9f25-istio-podinfo\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-zw5jl\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:00:20.796763 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:20.796747 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6183d0fa-f089-47f0-97a0-163160fc9f25-istiod-ca-cert\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-zw5jl\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:00:20.796900 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:20.796874 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-workload-socket\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-zw5jl\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:00:20.797059 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:20.796946 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-credential-socket\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-zw5jl\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:00:20.797059 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:20.796996 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvpw2\" (UniqueName: \"kubernetes.io/projected/6183d0fa-f089-47f0-97a0-163160fc9f25-kube-api-access-pvpw2\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-zw5jl\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:00:20.797059 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:20.797051 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-workload-certs\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-zw5jl\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:00:20.797289 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:20.797158 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-istio-data\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-zw5jl\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:00:20.797289 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:20.797191 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6183d0fa-f089-47f0-97a0-163160fc9f25-istio-token\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-zw5jl\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:00:20.797289 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:20.797203 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-workload-socket\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-zw5jl\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:00:20.797525 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:20.797300 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-credential-socket\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-zw5jl\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:00:20.797525 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:20.797501 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-istio-data\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-zw5jl\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:00:20.797645 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:20.797528 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6183d0fa-f089-47f0-97a0-163160fc9f25-istiod-ca-cert\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-zw5jl\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:00:20.797782 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:20.797761 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-workload-certs\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-zw5jl\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:00:20.799256 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:20.799216 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-istio-envoy\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-zw5jl\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:00:20.799667 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:20.799649 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6183d0fa-f089-47f0-97a0-163160fc9f25-istio-podinfo\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-zw5jl\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:00:20.822141 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:20.822110 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6183d0fa-f089-47f0-97a0-163160fc9f25-istio-token\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-zw5jl\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:00:20.822546 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:20.822515 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvpw2\" (UniqueName: \"kubernetes.io/projected/6183d0fa-f089-47f0-97a0-163160fc9f25-kube-api-access-pvpw2\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-zw5jl\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:00:20.946234 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:20.946160 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:00:21.285156 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:00:21.285124 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6183d0fa_f089_47f0_97a0_163160fc9f25.slice/crio-193f992fdfc9ad6a3ff527b4b557f8dd88b43bdbe649e26c4dbba885d1a1cace WatchSource:0}: Error finding container 193f992fdfc9ad6a3ff527b4b557f8dd88b43bdbe649e26c4dbba885d1a1cace: Status 404 returned error can't find the container with id 193f992fdfc9ad6a3ff527b4b557f8dd88b43bdbe649e26c4dbba885d1a1cace Nov 02 18:00:21.287773 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:21.287748 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl"] Nov 02 18:00:21.288753 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:21.288719 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:00:21.288850 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:21.288800 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:00:21.288850 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:21.288844 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:00:21.759444 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:21.759348 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" event={"ID":"6183d0fa-f089-47f0-97a0-163160fc9f25","Type":"ContainerStarted","Data":"033e549a2214d5dc6f831beaf5d2f08b85a6941e8511c9022f60186592b9896a"} Nov 02 18:00:21.759444 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:21.759419 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" event={"ID":"6183d0fa-f089-47f0-97a0-163160fc9f25","Type":"ContainerStarted","Data":"193f992fdfc9ad6a3ff527b4b557f8dd88b43bdbe649e26c4dbba885d1a1cace"} Nov 02 18:00:21.819452 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:21.819393 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" podStartSLOduration=1.819374147 podStartE2EDuration="1.819374147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:21.818355503 +0000 UTC m=+1839.513710977" watchObservedRunningTime="2026-11-02 18:00:21.819374147 +0000 UTC m=+1839.514729620" Nov 02 18:00:21.946700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:21.946663 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:00:22.960754 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:22.960456 4471 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 02 18:00:22.960754 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:22.960520 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" podUID="6183d0fa-f089-47f0-97a0-163160fc9f25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 02 18:00:23.946627 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:23.946595 4471 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 02 18:00:23.946789 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:23.946644 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" podUID="6183d0fa-f089-47f0-97a0-163160fc9f25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 02 18:00:24.433924 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:24.433885 4471 scope.go:122] "RemoveContainer" containerID="b927bb622aae8eccceb2b9ef032369ef04160a263a270759519b1d86f6eb0073" Nov 02 18:00:24.947125 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:24.947029 4471 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 02 18:00:24.947125 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:24.947091 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" podUID="6183d0fa-f089-47f0-97a0-163160fc9f25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 02 18:00:25.433690 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:25.433597 4471 scope.go:122] "RemoveContainer" containerID="fb05659122969b033d2a9ce1f396e3823561ebf004826a3e0dba62c1d3de79e8" Nov 02 18:00:25.433848 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:25.433813 4471 scope.go:122] "RemoveContainer" containerID="19462df9b74be4d63d3e8132c8a4631e8e248e9543aba0d7dea6afab3be67325" Nov 02 18:00:25.434599 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:25.434505 4471 scope.go:122] "RemoveContainer" containerID="b00d34d057953fe74795355fb34ef107bfa6eb88173a60fea0015c6afcd0c7de" Nov 02 18:00:25.800428 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:25.800398 4471 generic.go:356] "Generic (PLEG): container finished" podID="fab397a0-94f3-4ac9-9eea-36e8ca65ce1e" containerID="36db35287477002a21e9956b0f5c22c95aead17b2b8a78bd20be1e1b10e0d562" exitCode=60 Nov 02 18:00:25.800562 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:25.800468 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2" event={"ID":"fab397a0-94f3-4ac9-9eea-36e8ca65ce1e","Type":"ContainerDied","Data":"36db35287477002a21e9956b0f5c22c95aead17b2b8a78bd20be1e1b10e0d562"} Nov 02 18:00:25.800562 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:25.800529 4471 scope.go:122] "RemoveContainer" containerID="b927bb622aae8eccceb2b9ef032369ef04160a263a270759519b1d86f6eb0073" Nov 02 18:00:25.801077 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:25.801057 4471 scope.go:122] "RemoveContainer" containerID="36db35287477002a21e9956b0f5c22c95aead17b2b8a78bd20be1e1b10e0d562" Nov 02 18:00:25.801683 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:00:25.801592 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2_odh-ai-gateway-infra(fab397a0-94f3-4ac9-9eea-36e8ca65ce1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2" podUID="fab397a0-94f3-4ac9-9eea-36e8ca65ce1e" Nov 02 18:00:25.947099 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:25.947070 4471 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 02 18:00:25.947248 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:25.947121 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" podUID="6183d0fa-f089-47f0-97a0-163160fc9f25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 02 18:00:26.476427 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:26.474220 4471 scope.go:122] "RemoveContainer" containerID="61e83b7a81a510e2c854c957db784af701698689f7864c50b5645a9fa4d7b6c2" Nov 02 18:00:26.476427 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:26.475425 4471 scope.go:122] "RemoveContainer" containerID="7064f566b412caa86d08b6d717b6ef5798dac3229bfc4597aadba758c017f241" Nov 02 18:00:26.813751 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:26.813720 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9" event={"ID":"9206222b-4b98-4c20-b48c-dc6d671948c8","Type":"ContainerStarted","Data":"0d3a09032654d82fde659d1cc5d17bc322f45f527b542a7989d4f89064fb65b8"} Nov 02 18:00:26.815929 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:26.815905 4471 generic.go:356] "Generic (PLEG): container finished" podID="5976425a-97ac-4edd-97d9-4720709d298b" containerID="d8c3b15973c72d424ba0f94b46c0bf58cd71ee6bb34c7133ccbf0191dd4e6ce1" exitCode=60 Nov 02 18:00:26.816075 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:26.815980 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5" event={"ID":"5976425a-97ac-4edd-97d9-4720709d298b","Type":"ContainerDied","Data":"d8c3b15973c72d424ba0f94b46c0bf58cd71ee6bb34c7133ccbf0191dd4e6ce1"} Nov 02 18:00:26.816075 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:26.816054 4471 scope.go:122] "RemoveContainer" containerID="19462df9b74be4d63d3e8132c8a4631e8e248e9543aba0d7dea6afab3be67325" Nov 02 18:00:26.816540 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:26.816517 4471 scope.go:122] "RemoveContainer" containerID="d8c3b15973c72d424ba0f94b46c0bf58cd71ee6bb34c7133ccbf0191dd4e6ce1" Nov 02 18:00:26.817057 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:00:26.816920 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5_odh-ai-gateway-infra(5976425a-97ac-4edd-97d9-4720709d298b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5" podUID="5976425a-97ac-4edd-97d9-4720709d298b" Nov 02 18:00:26.818124 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:26.818099 4471 generic.go:356] "Generic (PLEG): container finished" podID="62e56641-f8f8-41be-bdcf-c31949402639" containerID="78901b5b7fd4a086103fc5f9fdcc77a915026ee3240f84120df6d4915d36b394" exitCode=60 Nov 02 18:00:26.818207 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:26.818170 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9" event={"ID":"62e56641-f8f8-41be-bdcf-c31949402639","Type":"ContainerDied","Data":"78901b5b7fd4a086103fc5f9fdcc77a915026ee3240f84120df6d4915d36b394"} Nov 02 18:00:26.818599 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:26.818581 4471 scope.go:122] "RemoveContainer" containerID="78901b5b7fd4a086103fc5f9fdcc77a915026ee3240f84120df6d4915d36b394" Nov 02 18:00:26.818983 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:00:26.818962 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9_odh-ai-gateway-infra(62e56641-f8f8-41be-bdcf-c31949402639)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9" podUID="62e56641-f8f8-41be-bdcf-c31949402639" Nov 02 18:00:26.906352 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:26.906324 4471 scope.go:122] "RemoveContainer" containerID="b00d34d057953fe74795355fb34ef107bfa6eb88173a60fea0015c6afcd0c7de" Nov 02 18:00:26.909854 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:26.909825 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2"] Nov 02 18:00:26.947387 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:26.947356 4471 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 02 18:00:26.947543 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:26.947418 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" podUID="6183d0fa-f089-47f0-97a0-163160fc9f25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 02 18:00:27.130109 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:27.130084 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2" Nov 02 18:00:27.208715 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:27.208680 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fab397a0-94f3-4ac9-9eea-36e8ca65ce1e-kube-api-access-fpbkx\" (UniqueName: \"kubernetes.io/projected/fab397a0-94f3-4ac9-9eea-36e8ca65ce1e-kube-api-access-fpbkx\") pod \"fab397a0-94f3-4ac9-9eea-36e8ca65ce1e\" (UID: \"fab397a0-94f3-4ac9-9eea-36e8ca65ce1e\") " Nov 02 18:00:27.208829 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:27.208772 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fab397a0-94f3-4ac9-9eea-36e8ca65ce1e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fab397a0-94f3-4ac9-9eea-36e8ca65ce1e-maas-api-service-ca\") pod \"fab397a0-94f3-4ac9-9eea-36e8ca65ce1e\" (UID: \"fab397a0-94f3-4ac9-9eea-36e8ca65ce1e\") " Nov 02 18:00:27.209275 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:27.209247 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fab397a0-94f3-4ac9-9eea-36e8ca65ce1e-maas-api-service-ca" pod "fab397a0-94f3-4ac9-9eea-36e8ca65ce1e" (UID: "fab397a0-94f3-4ac9-9eea-36e8ca65ce1e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:27.210014 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:27.209985 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fab397a0-94f3-4ac9-9eea-36e8ca65ce1e-maas-api-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:00:27.211272 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:27.211249 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab397a0-94f3-4ac9-9eea-36e8ca65ce1e-kube-api-access-fpbkx" pod "fab397a0-94f3-4ac9-9eea-36e8ca65ce1e" (UID: "fab397a0-94f3-4ac9-9eea-36e8ca65ce1e"). InnerVolumeSpecName "kube-api-access-fpbkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:27.311845 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:27.311765 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpbkx\" (UniqueName: \"kubernetes.io/projected/fab397a0-94f3-4ac9-9eea-36e8ca65ce1e-kube-api-access-fpbkx\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:00:27.433676 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:27.433643 4471 scope.go:122] "RemoveContainer" containerID="504c926100fc189ce4e98a99d0dec97faaf3388addcf2a3e6ddedce97c6b626f" Nov 02 18:00:27.832769 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:27.832739 4471 generic.go:356] "Generic (PLEG): container finished" podID="8afe383f-c164-43f8-bf0e-992689371bde" containerID="8ddc3a16e76020c9a7b6eda5cb6859f84369f859732ecd6bfa3a4dc3f695f5cc" exitCode=60 Nov 02 18:00:27.833197 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:27.832808 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x" event={"ID":"8afe383f-c164-43f8-bf0e-992689371bde","Type":"ContainerDied","Data":"8ddc3a16e76020c9a7b6eda5cb6859f84369f859732ecd6bfa3a4dc3f695f5cc"} Nov 02 18:00:27.833197 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:27.832863 4471 scope.go:122] "RemoveContainer" containerID="7064f566b412caa86d08b6d717b6ef5798dac3229bfc4597aadba758c017f241" Nov 02 18:00:27.833371 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:27.833353 4471 scope.go:122] "RemoveContainer" containerID="8ddc3a16e76020c9a7b6eda5cb6859f84369f859732ecd6bfa3a4dc3f695f5cc" Nov 02 18:00:27.834110 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:00:27.833788 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x_odh-ai-gateway-infra(8afe383f-c164-43f8-bf0e-992689371bde)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x" podUID="8afe383f-c164-43f8-bf0e-992689371bde" Nov 02 18:00:27.835091 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:27.835069 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp" event={"ID":"6eed364c-82b6-4fd8-8c04-4fef56994df6","Type":"ContainerStarted","Data":"90ae482baed5e97b43cbc6e810f04e26205f2a71e3d16ac9ce11623d8c0babef"} Nov 02 18:00:27.836453 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:27.836436 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2" Nov 02 18:00:27.836453 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:27.836443 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2" event={"ID":"fab397a0-94f3-4ac9-9eea-36e8ca65ce1e","Type":"ContainerDied","Data":"6ca03e144a89722da88069e42b1ce371dde83a271524c0c47b534dccde67cc24"} Nov 02 18:00:27.838383 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:27.838359 4471 generic.go:356] "Generic (PLEG): container finished" podID="9206222b-4b98-4c20-b48c-dc6d671948c8" containerID="0d3a09032654d82fde659d1cc5d17bc322f45f527b542a7989d4f89064fb65b8" exitCode=60 Nov 02 18:00:27.838475 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:27.838389 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9" event={"ID":"9206222b-4b98-4c20-b48c-dc6d671948c8","Type":"ContainerDied","Data":"0d3a09032654d82fde659d1cc5d17bc322f45f527b542a7989d4f89064fb65b8"} Nov 02 18:00:27.838818 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:27.838801 4471 scope.go:122] "RemoveContainer" containerID="0d3a09032654d82fde659d1cc5d17bc322f45f527b542a7989d4f89064fb65b8" Nov 02 18:00:27.839181 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:00:27.839158 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9_odh-ai-gateway-infra(9206222b-4b98-4c20-b48c-dc6d671948c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9" podUID="9206222b-4b98-4c20-b48c-dc6d671948c8" Nov 02 18:00:27.850726 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:27.850705 4471 scope.go:122] "RemoveContainer" containerID="36db35287477002a21e9956b0f5c22c95aead17b2b8a78bd20be1e1b10e0d562" Nov 02 18:00:27.889413 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:27.889384 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9"] Nov 02 18:00:27.889650 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:27.889622 4471 scope.go:122] "RemoveContainer" containerID="fb05659122969b033d2a9ce1f396e3823561ebf004826a3e0dba62c1d3de79e8" Nov 02 18:00:27.934079 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:27.934044 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9"] Nov 02 18:00:27.948434 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:27.948322 4471 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 02 18:00:27.948434 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:27.948364 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" podUID="6183d0fa-f089-47f0-97a0-163160fc9f25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 02 18:00:27.994391 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:27.993595 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5"] Nov 02 18:00:28.000671 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:28.000279 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2"] Nov 02 18:00:28.008712 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:28.008209 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894040-kcsp2"] Nov 02 18:00:28.434196 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:28.434144 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9" Nov 02 18:00:28.497994 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:28.497908 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fab397a0-94f3-4ac9-9eea-36e8ca65ce1e" path="/var/lib/kubelet/pods/fab397a0-94f3-4ac9-9eea-36e8ca65ce1e/volumes" Nov 02 18:00:28.527175 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:28.527138 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5" Nov 02 18:00:28.550387 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:28.550353 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5976425a-97ac-4edd-97d9-4720709d298b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5976425a-97ac-4edd-97d9-4720709d298b-maas-api-service-ca\") pod \"5976425a-97ac-4edd-97d9-4720709d298b\" (UID: \"5976425a-97ac-4edd-97d9-4720709d298b\") " Nov 02 18:00:28.550615 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:28.550601 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62e56641-f8f8-41be-bdcf-c31949402639-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e56641-f8f8-41be-bdcf-c31949402639-maas-api-service-ca\") pod \"62e56641-f8f8-41be-bdcf-c31949402639\" (UID: \"62e56641-f8f8-41be-bdcf-c31949402639\") " Nov 02 18:00:28.550760 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:28.550731 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5976425a-97ac-4edd-97d9-4720709d298b-maas-api-service-ca" pod "5976425a-97ac-4edd-97d9-4720709d298b" (UID: "5976425a-97ac-4edd-97d9-4720709d298b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:28.550817 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:28.550739 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62e56641-f8f8-41be-bdcf-c31949402639-kube-api-access-jmfdj\" (UniqueName: \"kubernetes.io/projected/62e56641-f8f8-41be-bdcf-c31949402639-kube-api-access-jmfdj\") pod \"62e56641-f8f8-41be-bdcf-c31949402639\" (UID: \"62e56641-f8f8-41be-bdcf-c31949402639\") " Nov 02 18:00:28.550871 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:28.550835 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5976425a-97ac-4edd-97d9-4720709d298b-kube-api-access-nkrx4\" (UniqueName: \"kubernetes.io/projected/5976425a-97ac-4edd-97d9-4720709d298b-kube-api-access-nkrx4\") pod \"5976425a-97ac-4edd-97d9-4720709d298b\" (UID: \"5976425a-97ac-4edd-97d9-4720709d298b\") " Nov 02 18:00:28.551406 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:28.551358 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62e56641-f8f8-41be-bdcf-c31949402639-maas-api-service-ca" pod "62e56641-f8f8-41be-bdcf-c31949402639" (UID: "62e56641-f8f8-41be-bdcf-c31949402639"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:28.552059 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:28.552035 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5976425a-97ac-4edd-97d9-4720709d298b-maas-api-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:00:28.552173 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:28.552065 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e56641-f8f8-41be-bdcf-c31949402639-maas-api-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:00:28.553562 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:28.553540 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5976425a-97ac-4edd-97d9-4720709d298b-kube-api-access-nkrx4" pod "5976425a-97ac-4edd-97d9-4720709d298b" (UID: "5976425a-97ac-4edd-97d9-4720709d298b"). InnerVolumeSpecName "kube-api-access-nkrx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:28.554179 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:28.553883 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e56641-f8f8-41be-bdcf-c31949402639-kube-api-access-jmfdj" pod "62e56641-f8f8-41be-bdcf-c31949402639" (UID: "62e56641-f8f8-41be-bdcf-c31949402639"). InnerVolumeSpecName "kube-api-access-jmfdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:28.654431 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:28.654394 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmfdj\" (UniqueName: \"kubernetes.io/projected/62e56641-f8f8-41be-bdcf-c31949402639-kube-api-access-jmfdj\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:00:28.654431 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:28.654430 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkrx4\" (UniqueName: \"kubernetes.io/projected/5976425a-97ac-4edd-97d9-4720709d298b-kube-api-access-nkrx4\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:00:28.850715 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:28.850678 4471 generic.go:356] "Generic (PLEG): container finished" podID="efdb807d-4f96-44d5-b40a-61206ca67e7f" containerID="fbc6fdc36e3c1a97c7bf2b4e704dec4af298bbc09af821081f8e813357b82adb" exitCode=60 Nov 02 18:00:28.851127 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:28.850751 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv" event={"ID":"efdb807d-4f96-44d5-b40a-61206ca67e7f","Type":"ContainerDied","Data":"fbc6fdc36e3c1a97c7bf2b4e704dec4af298bbc09af821081f8e813357b82adb"} Nov 02 18:00:28.851127 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:28.850809 4471 scope.go:122] "RemoveContainer" containerID="504c926100fc189ce4e98a99d0dec97faaf3388addcf2a3e6ddedce97c6b626f" Nov 02 18:00:28.851329 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:28.851285 4471 scope.go:122] "RemoveContainer" containerID="fbc6fdc36e3c1a97c7bf2b4e704dec4af298bbc09af821081f8e813357b82adb" Nov 02 18:00:28.852187 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:00:28.851827 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv_odh-ai-gateway-infra(efdb807d-4f96-44d5-b40a-61206ca67e7f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv" podUID="efdb807d-4f96-44d5-b40a-61206ca67e7f" Nov 02 18:00:28.854199 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:28.854181 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5" Nov 02 18:00:28.854346 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:28.854200 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5" event={"ID":"5976425a-97ac-4edd-97d9-4720709d298b","Type":"ContainerDied","Data":"cd267fe4b3201c8067c97e120200e46d9ec950c0a0996d792db07f94e70d8e38"} Nov 02 18:00:28.885347 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:28.876741 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9" event={"ID":"62e56641-f8f8-41be-bdcf-c31949402639","Type":"ContainerDied","Data":"a492921ebbdad1a8cabbde5c6d0a08cada87e0955d46846fde855de82aa1e463"} Nov 02 18:00:28.885347 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:28.876889 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9" Nov 02 18:00:28.894854 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:28.894815 4471 generic.go:356] "Generic (PLEG): container finished" podID="6eed364c-82b6-4fd8-8c04-4fef56994df6" containerID="90ae482baed5e97b43cbc6e810f04e26205f2a71e3d16ac9ce11623d8c0babef" exitCode=60 Nov 02 18:00:28.895037 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:28.894963 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp" event={"ID":"6eed364c-82b6-4fd8-8c04-4fef56994df6","Type":"ContainerDied","Data":"90ae482baed5e97b43cbc6e810f04e26205f2a71e3d16ac9ce11623d8c0babef"} Nov 02 18:00:28.895784 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:28.895745 4471 scope.go:122] "RemoveContainer" containerID="90ae482baed5e97b43cbc6e810f04e26205f2a71e3d16ac9ce11623d8c0babef" Nov 02 18:00:28.896457 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:00:28.896428 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp_odh-ai-gateway-infra(6eed364c-82b6-4fd8-8c04-4fef56994df6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp" podUID="6eed364c-82b6-4fd8-8c04-4fef56994df6" Nov 02 18:00:28.899690 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:28.899565 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x"] Nov 02 18:00:28.899840 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:28.899824 4471 scope.go:122] "RemoveContainer" containerID="d8c3b15973c72d424ba0f94b46c0bf58cd71ee6bb34c7133ccbf0191dd4e6ce1" Nov 02 18:00:28.917040 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:28.916919 4471 scope.go:122] "RemoveContainer" containerID="78901b5b7fd4a086103fc5f9fdcc77a915026ee3240f84120df6d4915d36b394" Nov 02 18:00:28.938001 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:28.937971 4471 scope.go:122] "RemoveContainer" containerID="61e83b7a81a510e2c854c957db784af701698689f7864c50b5645a9fa4d7b6c2" Nov 02 18:00:28.950552 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:28.949693 4471 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 02 18:00:28.951192 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:28.950776 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" podUID="6183d0fa-f089-47f0-97a0-163160fc9f25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 02 18:00:29.014070 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:29.014036 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp"] Nov 02 18:00:29.205359 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:29.204052 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9" Nov 02 18:00:29.226374 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:29.222773 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5"] Nov 02 18:00:29.231027 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:29.230981 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5c1da-29894040-t2gp5"] Nov 02 18:00:29.299726 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:29.299695 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9206222b-4b98-4c20-b48c-dc6d671948c8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9206222b-4b98-4c20-b48c-dc6d671948c8-maas-api-service-ca\") pod \"9206222b-4b98-4c20-b48c-dc6d671948c8\" (UID: \"9206222b-4b98-4c20-b48c-dc6d671948c8\") " Nov 02 18:00:29.300363 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:29.300335 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9206222b-4b98-4c20-b48c-dc6d671948c8-kube-api-access-8svz9\" (UniqueName: \"kubernetes.io/projected/9206222b-4b98-4c20-b48c-dc6d671948c8-kube-api-access-8svz9\") pod \"9206222b-4b98-4c20-b48c-dc6d671948c8\" (UID: \"9206222b-4b98-4c20-b48c-dc6d671948c8\") " Nov 02 18:00:29.301257 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:29.301226 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9206222b-4b98-4c20-b48c-dc6d671948c8-maas-api-service-ca" pod "9206222b-4b98-4c20-b48c-dc6d671948c8" (UID: "9206222b-4b98-4c20-b48c-dc6d671948c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:29.301741 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:29.301720 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9206222b-4b98-4c20-b48c-dc6d671948c8-maas-api-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:00:29.303659 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:29.303635 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9"] Nov 02 18:00:29.307497 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:29.305456 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9206222b-4b98-4c20-b48c-dc6d671948c8-kube-api-access-8svz9" pod "9206222b-4b98-4c20-b48c-dc6d671948c8" (UID: "9206222b-4b98-4c20-b48c-dc6d671948c8"). InnerVolumeSpecName "kube-api-access-8svz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:29.307614 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:29.307546 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x" Nov 02 18:00:29.311758 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:29.311733 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96768b-29894040-9x4k9"] Nov 02 18:00:29.405207 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:29.405113 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8afe383f-c164-43f8-bf0e-992689371bde-kube-api-access-xhlr2\" (UniqueName: \"kubernetes.io/projected/8afe383f-c164-43f8-bf0e-992689371bde-kube-api-access-xhlr2\") pod \"8afe383f-c164-43f8-bf0e-992689371bde\" (UID: \"8afe383f-c164-43f8-bf0e-992689371bde\") " Nov 02 18:00:29.405435 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:29.405297 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8afe383f-c164-43f8-bf0e-992689371bde-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8afe383f-c164-43f8-bf0e-992689371bde-maas-api-service-ca\") pod \"8afe383f-c164-43f8-bf0e-992689371bde\" (UID: \"8afe383f-c164-43f8-bf0e-992689371bde\") " Nov 02 18:00:29.436043 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:29.406205 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8svz9\" (UniqueName: \"kubernetes.io/projected/9206222b-4b98-4c20-b48c-dc6d671948c8-kube-api-access-8svz9\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:00:29.436043 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:29.406681 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8afe383f-c164-43f8-bf0e-992689371bde-maas-api-service-ca" pod "8afe383f-c164-43f8-bf0e-992689371bde" (UID: "8afe383f-c164-43f8-bf0e-992689371bde"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:29.436043 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:29.427024 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8afe383f-c164-43f8-bf0e-992689371bde-kube-api-access-xhlr2" pod "8afe383f-c164-43f8-bf0e-992689371bde" (UID: "8afe383f-c164-43f8-bf0e-992689371bde"). InnerVolumeSpecName "kube-api-access-xhlr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:29.436043 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:29.433494 4471 scope.go:122] "RemoveContainer" containerID="c1920c940c2c71739f33b54c1dd568153d8707c714975dd99e8212ae2e8216d2" Nov 02 18:00:29.510601 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:29.510078 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhlr2\" (UniqueName: \"kubernetes.io/projected/8afe383f-c164-43f8-bf0e-992689371bde-kube-api-access-xhlr2\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:00:29.510601 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:29.510099 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8afe383f-c164-43f8-bf0e-992689371bde-maas-api-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:00:29.918042 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:29.918001 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9" event={"ID":"9206222b-4b98-4c20-b48c-dc6d671948c8","Type":"ContainerDied","Data":"e9861d4cb142ec3a9f44ccf87b9d774d5a3e297a8afaf27dd69e5ebd742dbf68"} Nov 02 18:00:29.918657 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:29.918079 4471 scope.go:122] "RemoveContainer" containerID="0d3a09032654d82fde659d1cc5d17bc322f45f527b542a7989d4f89064fb65b8" Nov 02 18:00:29.918657 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:29.918181 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9" Nov 02 18:00:29.918657 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:29.918580 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv"] Nov 02 18:00:29.924581 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:29.924549 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x" event={"ID":"8afe383f-c164-43f8-bf0e-992689371bde","Type":"ContainerDied","Data":"4d8cb6742e4f6e32ffbe7e09d85306b811da153b5654e4ffa79431a834d41d35"} Nov 02 18:00:29.924677 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:29.924660 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x" Nov 02 18:00:29.940091 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:29.939818 4471 scope.go:122] "RemoveContainer" containerID="8ddc3a16e76020c9a7b6eda5cb6859f84369f859732ecd6bfa3a4dc3f695f5cc" Nov 02 18:00:29.948416 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:29.947329 4471 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 02 18:00:29.948416 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:29.947395 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" podUID="6183d0fa-f089-47f0-97a0-163160fc9f25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 02 18:00:30.033122 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.033091 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9"] Nov 02 18:00:30.043839 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.043806 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f6787-29894040-6nhv9"] Nov 02 18:00:30.205293 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.205137 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x"] Nov 02 18:00:30.250146 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.249838 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8a209f-29894040-hj22x"] Nov 02 18:00:30.381828 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.381791 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp" Nov 02 18:00:30.443941 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.443901 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eed364c-82b6-4fd8-8c04-4fef56994df6-kube-api-access-fl7f9\" (UniqueName: \"kubernetes.io/projected/6eed364c-82b6-4fd8-8c04-4fef56994df6-kube-api-access-fl7f9\") pod \"6eed364c-82b6-4fd8-8c04-4fef56994df6\" (UID: \"6eed364c-82b6-4fd8-8c04-4fef56994df6\") " Nov 02 18:00:30.444165 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.444048 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eed364c-82b6-4fd8-8c04-4fef56994df6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eed364c-82b6-4fd8-8c04-4fef56994df6-maas-api-service-ca\") pod \"6eed364c-82b6-4fd8-8c04-4fef56994df6\" (UID: \"6eed364c-82b6-4fd8-8c04-4fef56994df6\") " Nov 02 18:00:30.452155 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.449161 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eed364c-82b6-4fd8-8c04-4fef56994df6-maas-api-service-ca" pod "6eed364c-82b6-4fd8-8c04-4fef56994df6" (UID: "6eed364c-82b6-4fd8-8c04-4fef56994df6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:30.465401 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.464622 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eed364c-82b6-4fd8-8c04-4fef56994df6-kube-api-access-fl7f9" pod "6eed364c-82b6-4fd8-8c04-4fef56994df6" (UID: "6eed364c-82b6-4fd8-8c04-4fef56994df6"). InnerVolumeSpecName "kube-api-access-fl7f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:30.469182 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.469143 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5976425a-97ac-4edd-97d9-4720709d298b" path="/var/lib/kubelet/pods/5976425a-97ac-4edd-97d9-4720709d298b/volumes" Nov 02 18:00:30.474854 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.474820 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62e56641-f8f8-41be-bdcf-c31949402639" path="/var/lib/kubelet/pods/62e56641-f8f8-41be-bdcf-c31949402639/volumes" Nov 02 18:00:30.475856 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.475823 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8afe383f-c164-43f8-bf0e-992689371bde" path="/var/lib/kubelet/pods/8afe383f-c164-43f8-bf0e-992689371bde/volumes" Nov 02 18:00:30.476538 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.476513 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9206222b-4b98-4c20-b48c-dc6d671948c8" path="/var/lib/kubelet/pods/9206222b-4b98-4c20-b48c-dc6d671948c8/volumes" Nov 02 18:00:30.489630 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.489240 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv" Nov 02 18:00:30.545942 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.545905 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efdb807d-4f96-44d5-b40a-61206ca67e7f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efdb807d-4f96-44d5-b40a-61206ca67e7f-maas-api-service-ca\") pod \"efdb807d-4f96-44d5-b40a-61206ca67e7f\" (UID: \"efdb807d-4f96-44d5-b40a-61206ca67e7f\") " Nov 02 18:00:30.546148 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.546128 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efdb807d-4f96-44d5-b40a-61206ca67e7f-kube-api-access-pxkn2\" (UniqueName: \"kubernetes.io/projected/efdb807d-4f96-44d5-b40a-61206ca67e7f-kube-api-access-pxkn2\") pod \"efdb807d-4f96-44d5-b40a-61206ca67e7f\" (UID: \"efdb807d-4f96-44d5-b40a-61206ca67e7f\") " Nov 02 18:00:30.546540 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.546229 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efdb807d-4f96-44d5-b40a-61206ca67e7f-maas-api-service-ca" pod "efdb807d-4f96-44d5-b40a-61206ca67e7f" (UID: "efdb807d-4f96-44d5-b40a-61206ca67e7f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:30.547130 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.547108 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl7f9\" (UniqueName: \"kubernetes.io/projected/6eed364c-82b6-4fd8-8c04-4fef56994df6-kube-api-access-fl7f9\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:00:30.547181 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.547137 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eed364c-82b6-4fd8-8c04-4fef56994df6-maas-api-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:00:30.547181 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.547154 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efdb807d-4f96-44d5-b40a-61206ca67e7f-maas-api-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:00:30.548370 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.548346 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efdb807d-4f96-44d5-b40a-61206ca67e7f-kube-api-access-pxkn2" pod "efdb807d-4f96-44d5-b40a-61206ca67e7f" (UID: "efdb807d-4f96-44d5-b40a-61206ca67e7f"). InnerVolumeSpecName "kube-api-access-pxkn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:30.649007 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.648974 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxkn2\" (UniqueName: \"kubernetes.io/projected/efdb807d-4f96-44d5-b40a-61206ca67e7f-kube-api-access-pxkn2\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:00:30.938439 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.938405 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp" Nov 02 18:00:30.938439 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.938415 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp" event={"ID":"6eed364c-82b6-4fd8-8c04-4fef56994df6","Type":"ContainerDied","Data":"4fe5bc6d9b57d6a7a5992b07a007e07630690fc049082016cbfa6db4b19111d0"} Nov 02 18:00:30.938996 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.938483 4471 scope.go:122] "RemoveContainer" containerID="90ae482baed5e97b43cbc6e810f04e26205f2a71e3d16ac9ce11623d8c0babef" Nov 02 18:00:30.940090 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.940061 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv" Nov 02 18:00:30.940200 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.940109 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv" event={"ID":"efdb807d-4f96-44d5-b40a-61206ca67e7f","Type":"ContainerDied","Data":"438d1d240943d271cbefa9282b2c7c1fcd3c3ff1030e5342049df9aa0e758101"} Nov 02 18:00:30.949773 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.949080 4471 generic.go:356] "Generic (PLEG): container finished" podID="16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4" containerID="951b4407e885dd0581297c45828e8972b2f705584d7598f93dc915807ec8822c" exitCode=60 Nov 02 18:00:30.949773 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.949141 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr" event={"ID":"16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4","Type":"ContainerDied","Data":"951b4407e885dd0581297c45828e8972b2f705584d7598f93dc915807ec8822c"} Nov 02 18:00:30.949773 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.949680 4471 scope.go:122] "RemoveContainer" containerID="951b4407e885dd0581297c45828e8972b2f705584d7598f93dc915807ec8822c" Nov 02 18:00:30.950104 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:00:30.950084 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr_odh-ai-gateway-infra(16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr" podUID="16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4" Nov 02 18:00:30.951073 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.951053 4471 scope.go:122] "RemoveContainer" containerID="fbc6fdc36e3c1a97c7bf2b4e704dec4af298bbc09af821081f8e813357b82adb" Nov 02 18:00:30.954507 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.954220 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:00:30.954904 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.954884 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:00:30.955585 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.955567 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:00:30.963505 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:30.963486 4471 scope.go:122] "RemoveContainer" containerID="c1920c940c2c71739f33b54c1dd568153d8707c714975dd99e8212ae2e8216d2" Nov 02 18:00:31.099282 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:31.099194 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp"] Nov 02 18:00:31.104697 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:31.104661 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b913b2-29894040-7pwwp"] Nov 02 18:00:31.126937 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:31.126909 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv"] Nov 02 18:00:31.131437 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:31.131413 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b2fc9f-29894040-r84kv"] Nov 02 18:00:32.010106 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:32.010073 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr"] Nov 02 18:00:32.192397 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:32.192369 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr" Nov 02 18:00:32.281661 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:32.281625 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4-kube-api-access-v4k8m\" (UniqueName: \"kubernetes.io/projected/16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4-kube-api-access-v4k8m\") pod \"16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4\" (UID: \"16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4\") " Nov 02 18:00:32.281860 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:32.281685 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4-maas-api-service-ca\") pod \"16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4\" (UID: \"16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4\") " Nov 02 18:00:32.282208 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:32.282136 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4-maas-api-service-ca" pod "16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4" (UID: "16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:32.282843 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:32.282821 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4-maas-api-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:00:32.284578 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:32.284554 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4-kube-api-access-v4k8m" pod "16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4" (UID: "16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4"). InnerVolumeSpecName "kube-api-access-v4k8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:32.384425 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:32.384392 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4k8m\" (UniqueName: \"kubernetes.io/projected/16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4-kube-api-access-v4k8m\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:00:32.444236 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:32.444204 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eed364c-82b6-4fd8-8c04-4fef56994df6" path="/var/lib/kubelet/pods/6eed364c-82b6-4fd8-8c04-4fef56994df6/volumes" Nov 02 18:00:32.444773 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:32.444756 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efdb807d-4f96-44d5-b40a-61206ca67e7f" path="/var/lib/kubelet/pods/efdb807d-4f96-44d5-b40a-61206ca67e7f/volumes" Nov 02 18:00:32.974890 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:32.974859 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr" Nov 02 18:00:32.975104 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:32.974895 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr" event={"ID":"16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4","Type":"ContainerDied","Data":"3e3163c8e3c6e7325669a49d478b277d543be4dbcd20cdf6d9a146e8fa4d42fb"} Nov 02 18:00:32.975104 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:32.974960 4471 scope.go:122] "RemoveContainer" containerID="951b4407e885dd0581297c45828e8972b2f705584d7598f93dc915807ec8822c" Nov 02 18:00:33.042090 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:33.042056 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr"] Nov 02 18:00:33.050799 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:33.050749 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b91583-29894040-8mljr"] Nov 02 18:00:34.446036 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:34.445998 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4" path="/var/lib/kubelet/pods/16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4/volumes" Nov 02 18:00:39.200056 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:00:39.200017 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:00:47.338913 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:00:47.338875 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8"] Nov 02 18:01:02.370924 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:02.370882 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" podUID="198ca016-6343-4dda-999d-63a39d349f3a" containerName="main" containerID="cri-o://dd530edc0c36a79bedce01137e71c6666d838bc7df46d8c76dba442573acb775" gracePeriod=45 Nov 02 18:01:02.495783 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:02.495743 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" podUID="198ca016-6343-4dda-999d-63a39d349f3a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/ready\": dial tcp 10.133.0.142:8000: connect: connection refused" Nov 02 18:01:02.586404 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:02.586363 4471 generic.go:356] "Generic (PLEG): container finished" podID="198ca016-6343-4dda-999d-63a39d349f3a" containerID="dd530edc0c36a79bedce01137e71c6666d838bc7df46d8c76dba442573acb775" exitCode=0 Nov 02 18:01:02.586672 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:02.586648 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" event={"ID":"198ca016-6343-4dda-999d-63a39d349f3a","Type":"ContainerDied","Data":"dd530edc0c36a79bedce01137e71c6666d838bc7df46d8c76dba442573acb775"} Nov 02 18:01:02.679461 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:02.679433 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" Nov 02 18:01:02.851059 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:02.851026 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/198ca016-6343-4dda-999d-63a39d349f3a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/198ca016-6343-4dda-999d-63a39d349f3a-model-cache\") pod \"198ca016-6343-4dda-999d-63a39d349f3a\" (UID: \"198ca016-6343-4dda-999d-63a39d349f3a\") " Nov 02 18:01:02.851234 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:02.851161 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/198ca016-6343-4dda-999d-63a39d349f3a-home\" (UniqueName: \"kubernetes.io/empty-dir/198ca016-6343-4dda-999d-63a39d349f3a-home\") pod \"198ca016-6343-4dda-999d-63a39d349f3a\" (UID: \"198ca016-6343-4dda-999d-63a39d349f3a\") " Nov 02 18:01:02.851234 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:02.851208 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/198ca016-6343-4dda-999d-63a39d349f3a-tls-certs\" (UniqueName: \"kubernetes.io/secret/198ca016-6343-4dda-999d-63a39d349f3a-tls-certs\") pod \"198ca016-6343-4dda-999d-63a39d349f3a\" (UID: \"198ca016-6343-4dda-999d-63a39d349f3a\") " Nov 02 18:01:02.851330 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:02.851261 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/198ca016-6343-4dda-999d-63a39d349f3a-kube-api-access-g862n\" (UniqueName: \"kubernetes.io/projected/198ca016-6343-4dda-999d-63a39d349f3a-kube-api-access-g862n\") pod \"198ca016-6343-4dda-999d-63a39d349f3a\" (UID: \"198ca016-6343-4dda-999d-63a39d349f3a\") " Nov 02 18:01:02.851371 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:02.851291 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/198ca016-6343-4dda-999d-63a39d349f3a-model-cache" pod "198ca016-6343-4dda-999d-63a39d349f3a" (UID: "198ca016-6343-4dda-999d-63a39d349f3a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:02.851442 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:02.851423 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/198ca016-6343-4dda-999d-63a39d349f3a-home" pod "198ca016-6343-4dda-999d-63a39d349f3a" (UID: "198ca016-6343-4dda-999d-63a39d349f3a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:02.851505 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:02.851487 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/198ca016-6343-4dda-999d-63a39d349f3a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/198ca016-6343-4dda-999d-63a39d349f3a-tmp-dir\") pod \"198ca016-6343-4dda-999d-63a39d349f3a\" (UID: \"198ca016-6343-4dda-999d-63a39d349f3a\") " Nov 02 18:01:02.851624 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:02.851610 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/198ca016-6343-4dda-999d-63a39d349f3a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/198ca016-6343-4dda-999d-63a39d349f3a-dshm\") pod \"198ca016-6343-4dda-999d-63a39d349f3a\" (UID: \"198ca016-6343-4dda-999d-63a39d349f3a\") " Nov 02 18:01:02.851766 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:02.851746 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/198ca016-6343-4dda-999d-63a39d349f3a-tmp-dir" pod "198ca016-6343-4dda-999d-63a39d349f3a" (UID: "198ca016-6343-4dda-999d-63a39d349f3a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:02.852342 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:02.852299 4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/198ca016-6343-4dda-999d-63a39d349f3a-tmp-dir\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:01:02.852342 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:02.852341 4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/198ca016-6343-4dda-999d-63a39d349f3a-model-cache\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:01:02.852342 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:02.852349 4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/198ca016-6343-4dda-999d-63a39d349f3a-home\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:01:02.853710 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:02.853686 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/198ca016-6343-4dda-999d-63a39d349f3a-dshm" pod "198ca016-6343-4dda-999d-63a39d349f3a" (UID: "198ca016-6343-4dda-999d-63a39d349f3a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:02.853710 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:02.853696 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/198ca016-6343-4dda-999d-63a39d349f3a-tls-certs" pod "198ca016-6343-4dda-999d-63a39d349f3a" (UID: "198ca016-6343-4dda-999d-63a39d349f3a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:01:02.853861 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:02.853816 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/198ca016-6343-4dda-999d-63a39d349f3a-kube-api-access-g862n" pod "198ca016-6343-4dda-999d-63a39d349f3a" (UID: "198ca016-6343-4dda-999d-63a39d349f3a"). InnerVolumeSpecName "kube-api-access-g862n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:02.954523 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:02.954457 4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/198ca016-6343-4dda-999d-63a39d349f3a-dshm\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:01:02.954523 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:02.954483 4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/198ca016-6343-4dda-999d-63a39d349f3a-tls-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:01:02.954523 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:02.954496 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g862n\" (UniqueName: \"kubernetes.io/projected/198ca016-6343-4dda-999d-63a39d349f3a-kube-api-access-g862n\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:01:03.598671 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:03.598642 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" event={"ID":"198ca016-6343-4dda-999d-63a39d349f3a","Type":"ContainerDied","Data":"029ac69d576c521958922b57048248e7851213caf310ad1e20be9899983b848f"} Nov 02 18:01:03.598671 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:03.598662 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8" Nov 02 18:01:03.598671 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:03.598686 4471 scope.go:122] "RemoveContainer" containerID="dd530edc0c36a79bedce01137e71c6666d838bc7df46d8c76dba442573acb775" Nov 02 18:01:03.746291 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:03.744130 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8"] Nov 02 18:01:03.787756 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:03.787718 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-063f48/test-model-063f48-kserve-7b74d8c765-jdgd8"] Nov 02 18:01:04.444214 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:04.444179 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="198ca016-6343-4dda-999d-63a39d349f3a" path="/var/lib/kubelet/pods/198ca016-6343-4dda-999d-63a39d349f3a/volumes" Nov 02 18:01:05.686195 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:05.686163 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll"] Nov 02 18:01:05.686747 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:05.686382 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" podUID="636cf7f7-ba1a-4f54-9db0-6d8ad2909424" containerName="istio-proxy" containerID="cri-o://08ff90ee8d0ceace53f4f33083f4104ad89d3c9d5e2d23fec1ec3401760de26d" gracePeriod=30 Nov 02 18:01:08.007927 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:08.007888 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:01:08.008406 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:08.007963 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" podUID="636cf7f7-ba1a-4f54-9db0-6d8ad2909424" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:01:09.185561 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:01:09.185521 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:01:10.987216 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:10.987189 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 18:01:11.046422 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.046390 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istio-podinfo\") pod \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " Nov 02 18:01:11.046586 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.046568 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-workload-certs\") pod \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " Nov 02 18:01:11.046714 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.046696 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-workload-socket\") pod \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " Nov 02 18:01:11.046808 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.046746 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istiod-ca-cert\") pod \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " Nov 02 18:01:11.046808 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.046775 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istio-data\") pod \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " Nov 02 18:01:11.046975 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.046833 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istio-token\" (UniqueName: \"kubernetes.io/projected/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istio-token\") pod \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " Nov 02 18:01:11.046975 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.046946 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istio-envoy\") pod \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " Nov 02 18:01:11.047142 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.047080 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-kube-api-access-bmnll\" (UniqueName: \"kubernetes.io/projected/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-kube-api-access-bmnll\") pod \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " Nov 02 18:01:11.047227 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.047145 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-credential-socket\") pod \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\" (UID: \"636cf7f7-ba1a-4f54-9db0-6d8ad2909424\") " Nov 02 18:01:11.047848 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.047080 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-workload-socket" pod "636cf7f7-ba1a-4f54-9db0-6d8ad2909424" (UID: "636cf7f7-ba1a-4f54-9db0-6d8ad2909424"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:11.047848 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.047485 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istio-data" pod "636cf7f7-ba1a-4f54-9db0-6d8ad2909424" (UID: "636cf7f7-ba1a-4f54-9db0-6d8ad2909424"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:11.047848 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.047816 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istiod-ca-cert" pod "636cf7f7-ba1a-4f54-9db0-6d8ad2909424" (UID: "636cf7f7-ba1a-4f54-9db0-6d8ad2909424"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:01:11.048243 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.048184 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-credential-socket" pod "636cf7f7-ba1a-4f54-9db0-6d8ad2909424" (UID: "636cf7f7-ba1a-4f54-9db0-6d8ad2909424"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:11.048362 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.048246 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-workload-certs" pod "636cf7f7-ba1a-4f54-9db0-6d8ad2909424" (UID: "636cf7f7-ba1a-4f54-9db0-6d8ad2909424"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:11.048447 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.048384 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:01:11.048447 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.048408 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:01:11.048447 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.048423 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:01:11.049135 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.049109 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istio-podinfo" pod "636cf7f7-ba1a-4f54-9db0-6d8ad2909424" (UID: "636cf7f7-ba1a-4f54-9db0-6d8ad2909424"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:01:11.050457 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.050356 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istio-envoy" pod "636cf7f7-ba1a-4f54-9db0-6d8ad2909424" (UID: "636cf7f7-ba1a-4f54-9db0-6d8ad2909424"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:11.051419 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.051384 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istio-token" pod "636cf7f7-ba1a-4f54-9db0-6d8ad2909424" (UID: "636cf7f7-ba1a-4f54-9db0-6d8ad2909424"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:11.051531 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.051418 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-kube-api-access-bmnll" pod "636cf7f7-ba1a-4f54-9db0-6d8ad2909424" (UID: "636cf7f7-ba1a-4f54-9db0-6d8ad2909424"). InnerVolumeSpecName "kube-api-access-bmnll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:11.150035 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.150001 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:01:11.150035 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.150038 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:01:11.150228 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.150051 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmnll\" (UniqueName: \"kubernetes.io/projected/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-kube-api-access-bmnll\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:01:11.150228 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.150061 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:01:11.150228 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.150070 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:01:11.150228 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.150079 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/636cf7f7-ba1a-4f54-9db0-6d8ad2909424-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:01:11.718241 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.718204 4471 generic.go:356] "Generic (PLEG): container finished" podID="636cf7f7-ba1a-4f54-9db0-6d8ad2909424" containerID="08ff90ee8d0ceace53f4f33083f4104ad89d3c9d5e2d23fec1ec3401760de26d" exitCode=0 Nov 02 18:01:11.718429 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.718256 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" Nov 02 18:01:11.718429 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.718287 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" event={"ID":"636cf7f7-ba1a-4f54-9db0-6d8ad2909424","Type":"ContainerDied","Data":"08ff90ee8d0ceace53f4f33083f4104ad89d3c9d5e2d23fec1ec3401760de26d"} Nov 02 18:01:11.718429 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.718380 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll" event={"ID":"636cf7f7-ba1a-4f54-9db0-6d8ad2909424","Type":"ContainerDied","Data":"da81a82f5e2f15d2aae61fd8db3b18131a83e5c35a49fbe60ef6d5f6585cd921"} Nov 02 18:01:11.718429 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.718407 4471 scope.go:122] "RemoveContainer" containerID="08ff90ee8d0ceace53f4f33083f4104ad89d3c9d5e2d23fec1ec3401760de26d" Nov 02 18:01:11.730444 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.730423 4471 scope.go:122] "RemoveContainer" containerID="08ff90ee8d0ceace53f4f33083f4104ad89d3c9d5e2d23fec1ec3401760de26d" Nov 02 18:01:11.730715 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:01:11.730695 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08ff90ee8d0ceace53f4f33083f4104ad89d3c9d5e2d23fec1ec3401760de26d\": container with ID starting with 08ff90ee8d0ceace53f4f33083f4104ad89d3c9d5e2d23fec1ec3401760de26d not found: ID does not exist" containerID="08ff90ee8d0ceace53f4f33083f4104ad89d3c9d5e2d23fec1ec3401760de26d" Nov 02 18:01:11.730781 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.730727 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08ff90ee8d0ceace53f4f33083f4104ad89d3c9d5e2d23fec1ec3401760de26d"} err="failed to get container status \"08ff90ee8d0ceace53f4f33083f4104ad89d3c9d5e2d23fec1ec3401760de26d\": rpc error: code = NotFound desc = could not find container \"08ff90ee8d0ceace53f4f33083f4104ad89d3c9d5e2d23fec1ec3401760de26d\": container with ID starting with 08ff90ee8d0ceace53f4f33083f4104ad89d3c9d5e2d23fec1ec3401760de26d not found: ID does not exist" Nov 02 18:01:11.789079 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.789045 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll"] Nov 02 18:01:11.801193 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:11.801165 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-9x7ll"] Nov 02 18:01:12.466929 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:12.466897 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="636cf7f7-ba1a-4f54-9db0-6d8ad2909424" path="/var/lib/kubelet/pods/636cf7f7-ba1a-4f54-9db0-6d8ad2909424/volumes" Nov 02 18:01:14.824164 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:14.824127 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:01:17.609683 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:17.609648 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-063f48-578558f8d7-r97k5"] Nov 02 18:01:22.610457 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:22.610411 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-063f48-578558f8d7-r97k5" podUID="1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9" containerName="payload-pre-processing" containerID="cri-o://6eb740b19e330c59d7eca67580fdffb8026b9019b8cbb1a9b365f37b1d264cec" gracePeriod=25 Nov 02 18:01:28.513701 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:28.513659 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-063f48-578558f8d7-r97k5" podUID="1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:01:30.629871 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:30.629831 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb"] Nov 02 18:01:30.630409 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:30.630069 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" podUID="2b9ada91-7257-4ad9-952a-647e1163267d" containerName="istio-proxy" containerID="cri-o://9c5a818df6feb37c1e3e12fcc91db202531ddd81ca6c4fd2a2c55d2e7c1149fd" gracePeriod=30 Nov 02 18:01:32.580531 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:32.580497 4471 patch_prober.go:29] interesting pod/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:01:32.580998 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:32.580557 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" podUID="2b9ada91-7257-4ad9-952a-647e1163267d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:01:35.953682 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:35.953654 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 18:01:36.006638 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.006604 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b9ada91-7257-4ad9-952a-647e1163267d-istio-token\" (UniqueName: \"kubernetes.io/projected/2b9ada91-7257-4ad9-952a-647e1163267d-istio-token\") pod \"2b9ada91-7257-4ad9-952a-647e1163267d\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " Nov 02 18:01:36.006837 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.006795 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-istio-envoy\") pod \"2b9ada91-7257-4ad9-952a-647e1163267d\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " Nov 02 18:01:36.006915 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.006885 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b9ada91-7257-4ad9-952a-647e1163267d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b9ada91-7257-4ad9-952a-647e1163267d-istio-podinfo\") pod \"2b9ada91-7257-4ad9-952a-647e1163267d\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " Nov 02 18:01:36.007035 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.007016 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-workload-certs\") pod \"2b9ada91-7257-4ad9-952a-647e1163267d\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " Nov 02 18:01:36.007179 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.007124 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-istio-data\") pod \"2b9ada91-7257-4ad9-952a-647e1163267d\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " Nov 02 18:01:36.007398 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.007361 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-workload-socket\") pod \"2b9ada91-7257-4ad9-952a-647e1163267d\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " Nov 02 18:01:36.007489 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.007431 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b9ada91-7257-4ad9-952a-647e1163267d-kube-api-access-cxjg4\" (UniqueName: \"kubernetes.io/projected/2b9ada91-7257-4ad9-952a-647e1163267d-kube-api-access-cxjg4\") pod \"2b9ada91-7257-4ad9-952a-647e1163267d\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " Nov 02 18:01:36.007489 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.007441 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-workload-certs" pod "2b9ada91-7257-4ad9-952a-647e1163267d" (UID: "2b9ada91-7257-4ad9-952a-647e1163267d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:36.007602 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.007507 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-credential-socket\") pod \"2b9ada91-7257-4ad9-952a-647e1163267d\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " Nov 02 18:01:36.007602 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.007562 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b9ada91-7257-4ad9-952a-647e1163267d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b9ada91-7257-4ad9-952a-647e1163267d-istiod-ca-cert\") pod \"2b9ada91-7257-4ad9-952a-647e1163267d\" (UID: \"2b9ada91-7257-4ad9-952a-647e1163267d\") " Nov 02 18:01:36.007708 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.007645 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-istio-data" pod "2b9ada91-7257-4ad9-952a-647e1163267d" (UID: "2b9ada91-7257-4ad9-952a-647e1163267d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:36.008882 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.008512 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:01:36.008882 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.008540 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:01:36.009430 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.009336 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-credential-socket" pod "2b9ada91-7257-4ad9-952a-647e1163267d" (UID: "2b9ada91-7257-4ad9-952a-647e1163267d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:36.009430 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.009405 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-workload-socket" pod "2b9ada91-7257-4ad9-952a-647e1163267d" (UID: "2b9ada91-7257-4ad9-952a-647e1163267d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:36.009592 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.009423 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b9ada91-7257-4ad9-952a-647e1163267d-istio-token" pod "2b9ada91-7257-4ad9-952a-647e1163267d" (UID: "2b9ada91-7257-4ad9-952a-647e1163267d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:36.009895 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.009857 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b9ada91-7257-4ad9-952a-647e1163267d-istiod-ca-cert" pod "2b9ada91-7257-4ad9-952a-647e1163267d" (UID: "2b9ada91-7257-4ad9-952a-647e1163267d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:01:36.010101 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.010082 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b9ada91-7257-4ad9-952a-647e1163267d-kube-api-access-cxjg4" pod "2b9ada91-7257-4ad9-952a-647e1163267d" (UID: "2b9ada91-7257-4ad9-952a-647e1163267d"). InnerVolumeSpecName "kube-api-access-cxjg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:36.011834 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.011799 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b9ada91-7257-4ad9-952a-647e1163267d-istio-podinfo" pod "2b9ada91-7257-4ad9-952a-647e1163267d" (UID: "2b9ada91-7257-4ad9-952a-647e1163267d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:01:36.012071 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.012045 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-istio-envoy" pod "2b9ada91-7257-4ad9-952a-647e1163267d" (UID: "2b9ada91-7257-4ad9-952a-647e1163267d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:36.104625 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.104591 4471 generic.go:356] "Generic (PLEG): container finished" podID="2b9ada91-7257-4ad9-952a-647e1163267d" containerID="9c5a818df6feb37c1e3e12fcc91db202531ddd81ca6c4fd2a2c55d2e7c1149fd" exitCode=0 Nov 02 18:01:36.104815 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.104652 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" Nov 02 18:01:36.104815 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.104689 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" event={"ID":"2b9ada91-7257-4ad9-952a-647e1163267d","Type":"ContainerDied","Data":"9c5a818df6feb37c1e3e12fcc91db202531ddd81ca6c4fd2a2c55d2e7c1149fd"} Nov 02 18:01:36.104815 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.104760 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb" event={"ID":"2b9ada91-7257-4ad9-952a-647e1163267d","Type":"ContainerDied","Data":"cfb0270677fc126c46121e474a4004e5d3b456fb22c7caec13d48b643bad7133"} Nov 02 18:01:36.104815 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.104795 4471 scope.go:122] "RemoveContainer" containerID="9c5a818df6feb37c1e3e12fcc91db202531ddd81ca6c4fd2a2c55d2e7c1149fd" Nov 02 18:01:36.110124 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.110098 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxjg4\" (UniqueName: \"kubernetes.io/projected/2b9ada91-7257-4ad9-952a-647e1163267d-kube-api-access-cxjg4\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:01:36.110250 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.110128 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:01:36.110250 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.110142 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b9ada91-7257-4ad9-952a-647e1163267d-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:01:36.110250 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.110157 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b9ada91-7257-4ad9-952a-647e1163267d-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:01:36.110250 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.110171 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:01:36.110250 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.110183 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b9ada91-7257-4ad9-952a-647e1163267d-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:01:36.110250 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.110195 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9ada91-7257-4ad9-952a-647e1163267d-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:01:36.117784 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.117766 4471 scope.go:122] "RemoveContainer" containerID="6fbd95169d6bc1ab18fc0aa3ab752a9de8a50bb7074c63fff286b4750c4bf52a" Nov 02 18:01:36.127887 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.127869 4471 scope.go:122] "RemoveContainer" containerID="9c5a818df6feb37c1e3e12fcc91db202531ddd81ca6c4fd2a2c55d2e7c1149fd" Nov 02 18:01:36.128160 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:01:36.128141 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c5a818df6feb37c1e3e12fcc91db202531ddd81ca6c4fd2a2c55d2e7c1149fd\": container with ID starting with 9c5a818df6feb37c1e3e12fcc91db202531ddd81ca6c4fd2a2c55d2e7c1149fd not found: ID does not exist" containerID="9c5a818df6feb37c1e3e12fcc91db202531ddd81ca6c4fd2a2c55d2e7c1149fd" Nov 02 18:01:36.128208 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.128168 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c5a818df6feb37c1e3e12fcc91db202531ddd81ca6c4fd2a2c55d2e7c1149fd"} err="failed to get container status \"9c5a818df6feb37c1e3e12fcc91db202531ddd81ca6c4fd2a2c55d2e7c1149fd\": rpc error: code = NotFound desc = could not find container \"9c5a818df6feb37c1e3e12fcc91db202531ddd81ca6c4fd2a2c55d2e7c1149fd\": container with ID starting with 9c5a818df6feb37c1e3e12fcc91db202531ddd81ca6c4fd2a2c55d2e7c1149fd not found: ID does not exist" Nov 02 18:01:36.128208 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.128186 4471 scope.go:122] "RemoveContainer" containerID="6fbd95169d6bc1ab18fc0aa3ab752a9de8a50bb7074c63fff286b4750c4bf52a" Nov 02 18:01:36.128504 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:01:36.128487 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fbd95169d6bc1ab18fc0aa3ab752a9de8a50bb7074c63fff286b4750c4bf52a\": container with ID starting with 6fbd95169d6bc1ab18fc0aa3ab752a9de8a50bb7074c63fff286b4750c4bf52a not found: ID does not exist" containerID="6fbd95169d6bc1ab18fc0aa3ab752a9de8a50bb7074c63fff286b4750c4bf52a" Nov 02 18:01:36.128564 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.128510 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fbd95169d6bc1ab18fc0aa3ab752a9de8a50bb7074c63fff286b4750c4bf52a"} err="failed to get container status \"6fbd95169d6bc1ab18fc0aa3ab752a9de8a50bb7074c63fff286b4750c4bf52a\": rpc error: code = NotFound desc = could not find container \"6fbd95169d6bc1ab18fc0aa3ab752a9de8a50bb7074c63fff286b4750c4bf52a\": container with ID starting with 6fbd95169d6bc1ab18fc0aa3ab752a9de8a50bb7074c63fff286b4750c4bf52a not found: ID does not exist" Nov 02 18:01:36.166101 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.166023 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb"] Nov 02 18:01:36.174785 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.174754 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-mvdlb"] Nov 02 18:01:36.445103 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.445018 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b9ada91-7257-4ad9-952a-647e1163267d" path="/var/lib/kubelet/pods/2b9ada91-7257-4ad9-952a-647e1163267d/volumes" Nov 02 18:01:36.455881 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.455853 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv"] Nov 02 18:01:36.456803 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.456778 4471 cpu_manager.go:414] "containerMap: removing container" podUID="198ca016-6343-4dda-999d-63a39d349f3a" containerName="main" Nov 02 18:01:36.456803 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.456798 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="198ca016-6343-4dda-999d-63a39d349f3a" containerName="main" Nov 02 18:01:36.456964 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.456812 4471 cpu_manager.go:414] "containerMap: removing container" podUID="6eed364c-82b6-4fd8-8c04-4fef56994df6" containerName="cleanup" Nov 02 18:01:36.456964 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.456818 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eed364c-82b6-4fd8-8c04-4fef56994df6" containerName="cleanup" Nov 02 18:01:36.456964 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.456830 4471 cpu_manager.go:414] "containerMap: removing container" podUID="9206222b-4b98-4c20-b48c-dc6d671948c8" containerName="cleanup" Nov 02 18:01:36.456964 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.456836 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9206222b-4b98-4c20-b48c-dc6d671948c8" containerName="cleanup" Nov 02 18:01:36.456964 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.456847 4471 cpu_manager.go:414] "containerMap: removing container" podUID="9206222b-4b98-4c20-b48c-dc6d671948c8" containerName="cleanup" Nov 02 18:01:36.456964 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.456852 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9206222b-4b98-4c20-b48c-dc6d671948c8" containerName="cleanup" Nov 02 18:01:36.456964 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.456858 4471 cpu_manager.go:414] "containerMap: removing container" podUID="62e56641-f8f8-41be-bdcf-c31949402639" containerName="cleanup" Nov 02 18:01:36.456964 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.456863 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62e56641-f8f8-41be-bdcf-c31949402639" containerName="cleanup" Nov 02 18:01:36.456964 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.456872 4471 cpu_manager.go:414] "containerMap: removing container" podUID="fab397a0-94f3-4ac9-9eea-36e8ca65ce1e" containerName="cleanup" Nov 02 18:01:36.456964 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.456876 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fab397a0-94f3-4ac9-9eea-36e8ca65ce1e" containerName="cleanup" Nov 02 18:01:36.456964 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.456884 4471 cpu_manager.go:414] "containerMap: removing container" podUID="5976425a-97ac-4edd-97d9-4720709d298b" containerName="cleanup" Nov 02 18:01:36.456964 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.456889 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5976425a-97ac-4edd-97d9-4720709d298b" containerName="cleanup" Nov 02 18:01:36.456964 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.456898 4471 cpu_manager.go:414] "containerMap: removing container" podUID="8afe383f-c164-43f8-bf0e-992689371bde" containerName="cleanup" Nov 02 18:01:36.456964 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.456902 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8afe383f-c164-43f8-bf0e-992689371bde" containerName="cleanup" Nov 02 18:01:36.456964 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.456910 4471 cpu_manager.go:414] "containerMap: removing container" podUID="efdb807d-4f96-44d5-b40a-61206ca67e7f" containerName="cleanup" Nov 02 18:01:36.456964 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.456914 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efdb807d-4f96-44d5-b40a-61206ca67e7f" containerName="cleanup" Nov 02 18:01:36.456964 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.456925 4471 cpu_manager.go:414] "containerMap: removing container" podUID="16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4" containerName="cleanup" Nov 02 18:01:36.456964 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.456929 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4" containerName="cleanup" Nov 02 18:01:36.456964 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.456937 4471 cpu_manager.go:414] "containerMap: removing container" podUID="5976425a-97ac-4edd-97d9-4720709d298b" containerName="cleanup" Nov 02 18:01:36.456964 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.456941 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5976425a-97ac-4edd-97d9-4720709d298b" containerName="cleanup" Nov 02 18:01:36.456964 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.456948 4471 cpu_manager.go:414] "containerMap: removing container" podUID="5976425a-97ac-4edd-97d9-4720709d298b" containerName="cleanup" Nov 02 18:01:36.456964 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.456954 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5976425a-97ac-4edd-97d9-4720709d298b" containerName="cleanup" Nov 02 18:01:36.456964 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.456962 4471 cpu_manager.go:414] "containerMap: removing container" podUID="62e56641-f8f8-41be-bdcf-c31949402639" containerName="cleanup" Nov 02 18:01:36.456964 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.456967 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62e56641-f8f8-41be-bdcf-c31949402639" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.456977 4471 cpu_manager.go:414] "containerMap: removing container" podUID="8afe383f-c164-43f8-bf0e-992689371bde" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.456982 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8afe383f-c164-43f8-bf0e-992689371bde" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.456989 4471 cpu_manager.go:414] "containerMap: removing container" podUID="efdb807d-4f96-44d5-b40a-61206ca67e7f" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.456993 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efdb807d-4f96-44d5-b40a-61206ca67e7f" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457001 4471 cpu_manager.go:414] "containerMap: removing container" podUID="62e56641-f8f8-41be-bdcf-c31949402639" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457006 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62e56641-f8f8-41be-bdcf-c31949402639" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457015 4471 cpu_manager.go:414] "containerMap: removing container" podUID="fab397a0-94f3-4ac9-9eea-36e8ca65ce1e" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457019 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fab397a0-94f3-4ac9-9eea-36e8ca65ce1e" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457026 4471 cpu_manager.go:414] "containerMap: removing container" podUID="2b9ada91-7257-4ad9-952a-647e1163267d" containerName="istio-proxy" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457030 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b9ada91-7257-4ad9-952a-647e1163267d" containerName="istio-proxy" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457037 4471 cpu_manager.go:414] "containerMap: removing container" podUID="fab397a0-94f3-4ac9-9eea-36e8ca65ce1e" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457041 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fab397a0-94f3-4ac9-9eea-36e8ca65ce1e" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457048 4471 cpu_manager.go:414] "containerMap: removing container" podUID="efdb807d-4f96-44d5-b40a-61206ca67e7f" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457052 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efdb807d-4f96-44d5-b40a-61206ca67e7f" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457059 4471 cpu_manager.go:414] "containerMap: removing container" podUID="636cf7f7-ba1a-4f54-9db0-6d8ad2909424" containerName="istio-proxy" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457063 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="636cf7f7-ba1a-4f54-9db0-6d8ad2909424" containerName="istio-proxy" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457070 4471 cpu_manager.go:414] "containerMap: removing container" podUID="2b9ada91-7257-4ad9-952a-647e1163267d" containerName="istio-proxy" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457074 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b9ada91-7257-4ad9-952a-647e1163267d" containerName="istio-proxy" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457081 4471 cpu_manager.go:414] "containerMap: removing container" podUID="9206222b-4b98-4c20-b48c-dc6d671948c8" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457085 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9206222b-4b98-4c20-b48c-dc6d671948c8" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457106 4471 cpu_manager.go:414] "containerMap: removing container" podUID="6eed364c-82b6-4fd8-8c04-4fef56994df6" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457110 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eed364c-82b6-4fd8-8c04-4fef56994df6" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457117 4471 cpu_manager.go:414] "containerMap: removing container" podUID="16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457121 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457231 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="5976425a-97ac-4edd-97d9-4720709d298b" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457240 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="8afe383f-c164-43f8-bf0e-992689371bde" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457248 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="efdb807d-4f96-44d5-b40a-61206ca67e7f" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457254 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457260 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eed364c-82b6-4fd8-8c04-4fef56994df6" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457266 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="5976425a-97ac-4edd-97d9-4720709d298b" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457273 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="5976425a-97ac-4edd-97d9-4720709d298b" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457279 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="62e56641-f8f8-41be-bdcf-c31949402639" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457285 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457291 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="62e56641-f8f8-41be-bdcf-c31949402639" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457296 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="fab397a0-94f3-4ac9-9eea-36e8ca65ce1e" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457300 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="efdb807d-4f96-44d5-b40a-61206ca67e7f" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457322 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b9ada91-7257-4ad9-952a-647e1163267d" containerName="istio-proxy" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457327 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="fab397a0-94f3-4ac9-9eea-36e8ca65ce1e" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457332 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="636cf7f7-ba1a-4f54-9db0-6d8ad2909424" containerName="istio-proxy" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457338 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b9ada91-7257-4ad9-952a-647e1163267d" containerName="istio-proxy" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457344 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="9206222b-4b98-4c20-b48c-dc6d671948c8" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457350 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="8afe383f-c164-43f8-bf0e-992689371bde" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457360 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="198ca016-6343-4dda-999d-63a39d349f3a" containerName="main" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457365 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eed364c-82b6-4fd8-8c04-4fef56994df6" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457372 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="9206222b-4b98-4c20-b48c-dc6d671948c8" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457380 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="9206222b-4b98-4c20-b48c-dc6d671948c8" containerName="cleanup" Nov 02 18:01:36.457700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.457386 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="62e56641-f8f8-41be-bdcf-c31949402639" containerName="cleanup" Nov 02 18:01:36.461361 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.461343 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:01:36.493329 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.493278 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5fd3e0-openshift-default-dockercfg-kxlxh\"" Nov 02 18:01:36.501827 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.501795 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv"] Nov 02 18:01:36.506844 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.506818 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5"] Nov 02 18:01:36.508136 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.508116 4471 cpu_manager.go:414] "containerMap: removing container" podUID="6eed364c-82b6-4fd8-8c04-4fef56994df6" containerName="cleanup" Nov 02 18:01:36.508136 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.508136 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eed364c-82b6-4fd8-8c04-4fef56994df6" containerName="cleanup" Nov 02 18:01:36.508281 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.508169 4471 cpu_manager.go:414] "containerMap: removing container" podUID="16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4" containerName="cleanup" Nov 02 18:01:36.508281 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.508175 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4" containerName="cleanup" Nov 02 18:01:36.508281 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.508192 4471 cpu_manager.go:414] "containerMap: removing container" podUID="8afe383f-c164-43f8-bf0e-992689371bde" containerName="cleanup" Nov 02 18:01:36.508281 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.508198 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8afe383f-c164-43f8-bf0e-992689371bde" containerName="cleanup" Nov 02 18:01:36.508432 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.508336 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eed364c-82b6-4fd8-8c04-4fef56994df6" containerName="cleanup" Nov 02 18:01:36.508432 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.508352 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="16d9f56a-83c6-4143-a4d3-d96dc4c3f1f4" containerName="cleanup" Nov 02 18:01:36.508432 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.508359 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="efdb807d-4f96-44d5-b40a-61206ca67e7f" containerName="cleanup" Nov 02 18:01:36.508432 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.508365 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="fab397a0-94f3-4ac9-9eea-36e8ca65ce1e" containerName="cleanup" Nov 02 18:01:36.508432 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.508373 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="8afe383f-c164-43f8-bf0e-992689371bde" containerName="cleanup" Nov 02 18:01:36.513150 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.513127 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:01:36.518446 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.517947 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-workload-socket\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:01:36.518446 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.518073 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8210f11c-5046-4e0b-a3ca-d859918e76c7-istiod-ca-cert\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:01:36.518446 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.518152 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4hxm\" (UniqueName: \"kubernetes.io/projected/8210f11c-5046-4e0b-a3ca-d859918e76c7-kube-api-access-x4hxm\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:01:36.518446 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.518196 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-credential-socket\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:01:36.518446 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.518230 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-istio-envoy\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:01:36.518446 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.518291 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8210f11c-5046-4e0b-a3ca-d859918e76c7-istio-podinfo\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:01:36.518446 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.518355 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8210f11c-5046-4e0b-a3ca-d859918e76c7-istio-token\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:01:36.518446 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.518427 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-workload-certs\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:01:36.518937 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.518593 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-istio-data\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:01:36.542081 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.542051 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5"] Nov 02 18:01:36.620875 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.620838 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-workload-socket\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:01:36.621108 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.620891 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8210f11c-5046-4e0b-a3ca-d859918e76c7-istio-token\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:01:36.621108 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.621041 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-credential-socket\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:01:36.621260 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.621103 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-workload-certs\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:01:36.621378 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.621358 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-istio-data\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:01:36.621486 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.621466 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc4x9\" (UniqueName: \"kubernetes.io/projected/6ff6c042-8820-4706-bef8-e4cf2732a389-kube-api-access-hc4x9\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:01:36.621727 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.621706 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-workload-certs\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:01:36.621835 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.621759 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-istio-envoy\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:01:36.622584 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.621833 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-workload-certs\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:01:36.622584 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.621878 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-workload-socket\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:01:36.622584 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.621912 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-istio-data\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:01:36.622584 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.621940 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-istio-data\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:01:36.622584 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.622051 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff6c042-8820-4706-bef8-e4cf2732a389-istiod-ca-cert\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:01:36.622584 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.622088 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff6c042-8820-4706-bef8-e4cf2732a389-istio-token\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:01:36.622584 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.622126 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8210f11c-5046-4e0b-a3ca-d859918e76c7-istiod-ca-cert\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:01:36.622584 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.622142 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-workload-socket\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:01:36.622584 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.622208 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4hxm\" (UniqueName: \"kubernetes.io/projected/8210f11c-5046-4e0b-a3ca-d859918e76c7-kube-api-access-x4hxm\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:01:36.622584 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.622261 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-credential-socket\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:01:36.622584 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.622296 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff6c042-8820-4706-bef8-e4cf2732a389-istio-podinfo\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:01:36.622584 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.622346 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-istio-envoy\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:01:36.622584 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.622416 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8210f11c-5046-4e0b-a3ca-d859918e76c7-istio-podinfo\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:01:36.623273 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.622816 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8210f11c-5046-4e0b-a3ca-d859918e76c7-istiod-ca-cert\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:01:36.623273 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.623076 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-credential-socket\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:01:36.625099 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.625075 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8210f11c-5046-4e0b-a3ca-d859918e76c7-istio-podinfo\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:01:36.625374 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.625353 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-istio-envoy\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:01:36.658694 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.658662 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8210f11c-5046-4e0b-a3ca-d859918e76c7-istio-token\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:01:36.659146 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.659126 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4hxm\" (UniqueName: \"kubernetes.io/projected/8210f11c-5046-4e0b-a3ca-d859918e76c7-kube-api-access-x4hxm\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:01:36.724351 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.724170 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-workload-socket\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:01:36.724502 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.724352 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-credential-socket\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:01:36.724552 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.724518 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc4x9\" (UniqueName: \"kubernetes.io/projected/6ff6c042-8820-4706-bef8-e4cf2732a389-kube-api-access-hc4x9\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:01:36.724594 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.724567 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-workload-socket\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:01:36.724651 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.724572 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-istio-envoy\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:01:36.724651 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.724641 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-workload-certs\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:01:36.724734 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.724659 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-credential-socket\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:01:36.724734 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.724680 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-istio-data\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:01:36.724804 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.724760 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff6c042-8820-4706-bef8-e4cf2732a389-istiod-ca-cert\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:01:36.724901 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.724805 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff6c042-8820-4706-bef8-e4cf2732a389-istio-token\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:01:36.724901 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.724888 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-workload-certs\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:01:36.725068 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.725044 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff6c042-8820-4706-bef8-e4cf2732a389-istio-podinfo\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:01:36.725129 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.725068 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-istio-data\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:01:36.725474 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.725451 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff6c042-8820-4706-bef8-e4cf2732a389-istiod-ca-cert\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:01:36.726777 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.726754 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-istio-envoy\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:01:36.727430 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.727412 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff6c042-8820-4706-bef8-e4cf2732a389-istio-podinfo\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:01:36.758133 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.758110 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff6c042-8820-4706-bef8-e4cf2732a389-istio-token\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:01:36.758315 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.758282 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc4x9\" (UniqueName: \"kubernetes.io/projected/6ff6c042-8820-4706-bef8-e4cf2732a389-kube-api-access-hc4x9\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:01:36.777218 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.777198 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:01:36.849499 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:36.849234 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:01:37.097948 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:01:37.097918 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8210f11c_5046_4e0b_a3ca_d859918e76c7.slice/crio-476af9d9e74d5f29f05c3c1122701d9e2496acdb7faf27072385352f929181b3 WatchSource:0}: Error finding container 476af9d9e74d5f29f05c3c1122701d9e2496acdb7faf27072385352f929181b3: Status 404 returned error can't find the container with id 476af9d9e74d5f29f05c3c1122701d9e2496acdb7faf27072385352f929181b3 Nov 02 18:01:37.101910 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:37.101866 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:01:37.102084 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:37.102057 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:01:37.102180 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:37.102161 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:01:37.104633 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:37.104610 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv"] Nov 02 18:01:37.122384 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:37.122355 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" event={"ID":"8210f11c-5046-4e0b-a3ca-d859918e76c7","Type":"ContainerStarted","Data":"476af9d9e74d5f29f05c3c1122701d9e2496acdb7faf27072385352f929181b3"} Nov 02 18:01:37.174076 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:01:37.174050 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff6c042_8820_4706_bef8_e4cf2732a389.slice/crio-df341392f4bfe7d5960f4a4c2c4f06ad3a507999d22df161d60907d7b6a4f175 WatchSource:0}: Error finding container df341392f4bfe7d5960f4a4c2c4f06ad3a507999d22df161d60907d7b6a4f175: Status 404 returned error can't find the container with id df341392f4bfe7d5960f4a4c2c4f06ad3a507999d22df161d60907d7b6a4f175 Nov 02 18:01:37.177132 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:37.177108 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5"] Nov 02 18:01:37.177597 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:37.177564 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:01:37.177693 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:37.177635 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:01:37.177693 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:37.177665 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:01:38.135726 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:38.135689 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" event={"ID":"6ff6c042-8820-4706-bef8-e4cf2732a389","Type":"ContainerStarted","Data":"7ffd35bc256878511d42c8959e6a6f834cd9d83ff47ab3a3ff635817057724ae"} Nov 02 18:01:38.136210 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:38.135780 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" event={"ID":"6ff6c042-8820-4706-bef8-e4cf2732a389","Type":"ContainerStarted","Data":"df341392f4bfe7d5960f4a4c2c4f06ad3a507999d22df161d60907d7b6a4f175"} Nov 02 18:01:38.137886 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:38.137857 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" event={"ID":"8210f11c-5046-4e0b-a3ca-d859918e76c7","Type":"ContainerStarted","Data":"6fb440c60b1ae9dabe51a4b1c3170b9d9f39b060fa910f0af37932cb43dacdd8"} Nov 02 18:01:38.200111 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:38.200062 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" podStartSLOduration=2.200047289 podStartE2EDuration="2.200047289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:38.194949055 +0000 UTC m=+1915.890304552" watchObservedRunningTime="2026-11-02 18:01:38.200047289 +0000 UTC m=+1915.895402836" Nov 02 18:01:38.250999 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:38.250945 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" podStartSLOduration=2.250931067 podStartE2EDuration="2.250931067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:38.245241376 +0000 UTC m=+1915.940596850" watchObservedRunningTime="2026-11-02 18:01:38.250931067 +0000 UTC m=+1915.946286540" Nov 02 18:01:38.513256 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:38.513222 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-063f48-578558f8d7-r97k5" podUID="1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:01:38.787388 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:38.786727 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:01:38.809332 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:38.807438 4471 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 02 18:01:38.809332 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:38.807503 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" podUID="8210f11c-5046-4e0b-a3ca-d859918e76c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 02 18:01:38.850367 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:38.850332 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:01:38.851858 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:38.851837 4471 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 02 18:01:38.851950 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:38.851887 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" podUID="6ff6c042-8820-4706-bef8-e4cf2732a389" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 02 18:01:39.117280 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:39.117186 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c"] Nov 02 18:01:39.123065 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:39.122731 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" Nov 02 18:01:39.150375 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:39.150346 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c"] Nov 02 18:01:39.153967 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:39.153941 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-96768b\"/\"default-dockercfg-qbcxx\"" Nov 02 18:01:39.154139 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:39.154032 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-96768b\"/\"test-model-96768b-kserve-self-signed-certs\"" Nov 02 18:01:39.204330 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:39.179531 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-96768b\"/\"kube-root-ca.crt\"" Nov 02 18:01:39.213333 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:39.180473 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-96768b\"/\"openshift-service-ca.crt\"" Nov 02 18:01:39.225630 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:39.224888 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fad2db08-e940-4823-99ee-9b3fb1346140-dshm\") pod \"test-model-96768b-kserve-74d57ffdc-l9t5c\" (UID: \"fad2db08-e940-4823-99ee-9b3fb1346140\") " pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" Nov 02 18:01:39.225630 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:39.224979 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fad2db08-e940-4823-99ee-9b3fb1346140-tls-certs\") pod \"test-model-96768b-kserve-74d57ffdc-l9t5c\" (UID: \"fad2db08-e940-4823-99ee-9b3fb1346140\") " pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" Nov 02 18:01:39.225630 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:39.225193 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gccq9\" (UniqueName: \"kubernetes.io/projected/fad2db08-e940-4823-99ee-9b3fb1346140-kube-api-access-gccq9\") pod \"test-model-96768b-kserve-74d57ffdc-l9t5c\" (UID: \"fad2db08-e940-4823-99ee-9b3fb1346140\") " pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" Nov 02 18:01:39.225630 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:39.225293 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fad2db08-e940-4823-99ee-9b3fb1346140-home\") pod \"test-model-96768b-kserve-74d57ffdc-l9t5c\" (UID: \"fad2db08-e940-4823-99ee-9b3fb1346140\") " pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" Nov 02 18:01:39.225630 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:39.225373 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fad2db08-e940-4823-99ee-9b3fb1346140-model-cache\") pod \"test-model-96768b-kserve-74d57ffdc-l9t5c\" (UID: \"fad2db08-e940-4823-99ee-9b3fb1346140\") " pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" Nov 02 18:01:39.225630 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:39.225421 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fad2db08-e940-4823-99ee-9b3fb1346140-tmp-dir\") pod \"test-model-96768b-kserve-74d57ffdc-l9t5c\" (UID: \"fad2db08-e940-4823-99ee-9b3fb1346140\") " pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" Nov 02 18:01:39.308875 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:01:39.308835 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:01:39.326821 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:39.326776 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gccq9\" (UniqueName: \"kubernetes.io/projected/fad2db08-e940-4823-99ee-9b3fb1346140-kube-api-access-gccq9\") pod \"test-model-96768b-kserve-74d57ffdc-l9t5c\" (UID: \"fad2db08-e940-4823-99ee-9b3fb1346140\") " pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" Nov 02 18:01:39.327055 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:39.326903 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fad2db08-e940-4823-99ee-9b3fb1346140-home\") pod \"test-model-96768b-kserve-74d57ffdc-l9t5c\" (UID: \"fad2db08-e940-4823-99ee-9b3fb1346140\") " pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" Nov 02 18:01:39.327055 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:39.326995 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fad2db08-e940-4823-99ee-9b3fb1346140-model-cache\") pod \"test-model-96768b-kserve-74d57ffdc-l9t5c\" (UID: \"fad2db08-e940-4823-99ee-9b3fb1346140\") " pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" Nov 02 18:01:39.327185 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:39.327062 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fad2db08-e940-4823-99ee-9b3fb1346140-tmp-dir\") pod \"test-model-96768b-kserve-74d57ffdc-l9t5c\" (UID: \"fad2db08-e940-4823-99ee-9b3fb1346140\") " pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" Nov 02 18:01:39.327276 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:39.327198 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fad2db08-e940-4823-99ee-9b3fb1346140-dshm\") pod \"test-model-96768b-kserve-74d57ffdc-l9t5c\" (UID: \"fad2db08-e940-4823-99ee-9b3fb1346140\") " pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" Nov 02 18:01:39.327545 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:39.327518 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fad2db08-e940-4823-99ee-9b3fb1346140-tls-certs\") pod \"test-model-96768b-kserve-74d57ffdc-l9t5c\" (UID: \"fad2db08-e940-4823-99ee-9b3fb1346140\") " pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" Nov 02 18:01:39.328035 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:39.328009 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fad2db08-e940-4823-99ee-9b3fb1346140-tmp-dir\") pod \"test-model-96768b-kserve-74d57ffdc-l9t5c\" (UID: \"fad2db08-e940-4823-99ee-9b3fb1346140\") " pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" Nov 02 18:01:39.328035 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:39.327962 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fad2db08-e940-4823-99ee-9b3fb1346140-home\") pod \"test-model-96768b-kserve-74d57ffdc-l9t5c\" (UID: \"fad2db08-e940-4823-99ee-9b3fb1346140\") " pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" Nov 02 18:01:39.328193 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:39.327537 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fad2db08-e940-4823-99ee-9b3fb1346140-model-cache\") pod \"test-model-96768b-kserve-74d57ffdc-l9t5c\" (UID: \"fad2db08-e940-4823-99ee-9b3fb1346140\") " pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" Nov 02 18:01:39.331117 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:39.331092 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fad2db08-e940-4823-99ee-9b3fb1346140-tls-certs\") pod \"test-model-96768b-kserve-74d57ffdc-l9t5c\" (UID: \"fad2db08-e940-4823-99ee-9b3fb1346140\") " pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" Nov 02 18:01:39.331357 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:39.331319 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fad2db08-e940-4823-99ee-9b3fb1346140-dshm\") pod \"test-model-96768b-kserve-74d57ffdc-l9t5c\" (UID: \"fad2db08-e940-4823-99ee-9b3fb1346140\") " pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" Nov 02 18:01:39.353052 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:39.353024 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gccq9\" (UniqueName: \"kubernetes.io/projected/fad2db08-e940-4823-99ee-9b3fb1346140-kube-api-access-gccq9\") pod \"test-model-96768b-kserve-74d57ffdc-l9t5c\" (UID: \"fad2db08-e940-4823-99ee-9b3fb1346140\") " pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" Nov 02 18:01:39.443421 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:39.443342 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" Nov 02 18:01:39.787535 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:39.787433 4471 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 02 18:01:39.787535 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:39.787517 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" podUID="8210f11c-5046-4e0b-a3ca-d859918e76c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 02 18:01:39.855273 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:39.855236 4471 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 02 18:01:39.855487 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:39.855296 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" podUID="6ff6c042-8820-4706-bef8-e4cf2732a389" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 02 18:01:40.448458 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:01:40.448428 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfad2db08_e940_4823_99ee_9b3fb1346140.slice/crio-7777675afb8acd43cea6ef25c90a1d256c86a21ad91cc4e6a82b38458942a78a WatchSource:0}: Error finding container 7777675afb8acd43cea6ef25c90a1d256c86a21ad91cc4e6a82b38458942a78a: Status 404 returned error can't find the container with id 7777675afb8acd43cea6ef25c90a1d256c86a21ad91cc4e6a82b38458942a78a Nov 02 18:01:40.452089 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:40.451950 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c"] Nov 02 18:01:40.803359 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:40.790482 4471 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 02 18:01:40.803359 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:40.790575 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" podUID="8210f11c-5046-4e0b-a3ca-d859918e76c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 02 18:01:40.829799 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:40.829763 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx"] Nov 02 18:01:40.899328 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:40.859561 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx" Nov 02 18:01:40.899328 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:40.874489 4471 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 02 18:01:40.899328 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:40.874602 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" podUID="6ff6c042-8820-4706-bef8-e4cf2732a389" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 02 18:01:40.899328 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:40.896717 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5fd3e0\"/\"openshift-service-ca.crt\"" Nov 02 18:01:40.899328 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:40.896911 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5fd3e0\"/\"kube-root-ca.crt\"" Nov 02 18:01:40.928227 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:40.910798 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5fd3e0\"/\"default-dockercfg-gnnzc\"" Nov 02 18:01:40.928227 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:40.911040 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5fd3e0\"/\"test-model-5fd3e0-kserve-self-signed-certs\"" Nov 02 18:01:40.928227 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:40.927670 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx"] Nov 02 18:01:41.031948 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:41.031909 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b6xq\" (UniqueName: \"kubernetes.io/projected/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-kube-api-access-2b6xq\") pod \"test-model-5fd3e0-kserve-56cc48b75c-dptkx\" (UID: \"9b1845b8-af82-4fd6-8950-cfccf0c9ea8e\") " pod="ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx" Nov 02 18:01:41.032124 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:41.031970 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-tls-certs\") pod \"test-model-5fd3e0-kserve-56cc48b75c-dptkx\" (UID: \"9b1845b8-af82-4fd6-8950-cfccf0c9ea8e\") " pod="ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx" Nov 02 18:01:41.032124 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:41.032044 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-model-cache\") pod \"test-model-5fd3e0-kserve-56cc48b75c-dptkx\" (UID: \"9b1845b8-af82-4fd6-8950-cfccf0c9ea8e\") " pod="ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx" Nov 02 18:01:41.032277 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:41.032260 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-dshm\") pod \"test-model-5fd3e0-kserve-56cc48b75c-dptkx\" (UID: \"9b1845b8-af82-4fd6-8950-cfccf0c9ea8e\") " pod="ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx" Nov 02 18:01:41.032337 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:41.032298 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-tmp-dir\") pod \"test-model-5fd3e0-kserve-56cc48b75c-dptkx\" (UID: \"9b1845b8-af82-4fd6-8950-cfccf0c9ea8e\") " pod="ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx" Nov 02 18:01:41.032392 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:41.032379 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-home\") pod \"test-model-5fd3e0-kserve-56cc48b75c-dptkx\" (UID: \"9b1845b8-af82-4fd6-8950-cfccf0c9ea8e\") " pod="ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx" Nov 02 18:01:41.162331 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:41.161116 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-home\") pod \"test-model-5fd3e0-kserve-56cc48b75c-dptkx\" (UID: \"9b1845b8-af82-4fd6-8950-cfccf0c9ea8e\") " pod="ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx" Nov 02 18:01:41.162331 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:41.161339 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b6xq\" (UniqueName: \"kubernetes.io/projected/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-kube-api-access-2b6xq\") pod \"test-model-5fd3e0-kserve-56cc48b75c-dptkx\" (UID: \"9b1845b8-af82-4fd6-8950-cfccf0c9ea8e\") " pod="ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx" Nov 02 18:01:41.162331 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:41.161389 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-tls-certs\") pod \"test-model-5fd3e0-kserve-56cc48b75c-dptkx\" (UID: \"9b1845b8-af82-4fd6-8950-cfccf0c9ea8e\") " pod="ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx" Nov 02 18:01:41.162331 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:41.161419 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-model-cache\") pod \"test-model-5fd3e0-kserve-56cc48b75c-dptkx\" (UID: \"9b1845b8-af82-4fd6-8950-cfccf0c9ea8e\") " pod="ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx" Nov 02 18:01:41.162331 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:41.161659 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-dshm\") pod \"test-model-5fd3e0-kserve-56cc48b75c-dptkx\" (UID: \"9b1845b8-af82-4fd6-8950-cfccf0c9ea8e\") " pod="ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx" Nov 02 18:01:41.162331 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:41.161703 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-tmp-dir\") pod \"test-model-5fd3e0-kserve-56cc48b75c-dptkx\" (UID: \"9b1845b8-af82-4fd6-8950-cfccf0c9ea8e\") " pod="ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx" Nov 02 18:01:41.162331 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:41.162178 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-tmp-dir\") pod \"test-model-5fd3e0-kserve-56cc48b75c-dptkx\" (UID: \"9b1845b8-af82-4fd6-8950-cfccf0c9ea8e\") " pod="ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx" Nov 02 18:01:41.213933 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:41.213897 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-model-cache\") pod \"test-model-5fd3e0-kserve-56cc48b75c-dptkx\" (UID: \"9b1845b8-af82-4fd6-8950-cfccf0c9ea8e\") " pod="ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx" Nov 02 18:01:41.248329 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:41.245599 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-home\") pod \"test-model-5fd3e0-kserve-56cc48b75c-dptkx\" (UID: \"9b1845b8-af82-4fd6-8950-cfccf0c9ea8e\") " pod="ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx" Nov 02 18:01:41.275336 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:41.270226 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-tls-certs\") pod \"test-model-5fd3e0-kserve-56cc48b75c-dptkx\" (UID: \"9b1845b8-af82-4fd6-8950-cfccf0c9ea8e\") " pod="ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx" Nov 02 18:01:41.330380 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:41.307034 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b6xq\" (UniqueName: \"kubernetes.io/projected/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-kube-api-access-2b6xq\") pod \"test-model-5fd3e0-kserve-56cc48b75c-dptkx\" (UID: \"9b1845b8-af82-4fd6-8950-cfccf0c9ea8e\") " pod="ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx" Nov 02 18:01:41.338328 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:41.335488 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-dshm\") pod \"test-model-5fd3e0-kserve-56cc48b75c-dptkx\" (UID: \"9b1845b8-af82-4fd6-8950-cfccf0c9ea8e\") " pod="ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx" Nov 02 18:01:41.538076 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:41.537985 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx" Nov 02 18:01:41.543514 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:41.543147 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" event={"ID":"fad2db08-e940-4823-99ee-9b3fb1346140","Type":"ContainerStarted","Data":"cd93bcb19130cbbe6b61530213047ebdc8dd4ac371b9b9485bc6a8c12b362291"} Nov 02 18:01:41.543514 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:41.543202 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" event={"ID":"fad2db08-e940-4823-99ee-9b3fb1346140","Type":"ContainerStarted","Data":"7777675afb8acd43cea6ef25c90a1d256c86a21ad91cc4e6a82b38458942a78a"} Nov 02 18:01:41.644996 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:41.644941 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" podStartSLOduration=2.644926672 podStartE2EDuration="2.644926672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:41.643540797 +0000 UTC m=+1919.338896270" watchObservedRunningTime="2026-11-02 18:01:41.644926672 +0000 UTC m=+1919.340282145" Nov 02 18:01:41.788390 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:41.785821 4471 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 02 18:01:41.788390 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:41.785913 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" podUID="8210f11c-5046-4e0b-a3ca-d859918e76c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 02 18:01:41.869454 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:41.864127 4471 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 02 18:01:41.869758 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:41.869716 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" podUID="6ff6c042-8820-4706-bef8-e4cf2732a389" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 02 18:01:42.448488 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:42.448457 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" Nov 02 18:01:42.513037 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:42.513001 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx"] Nov 02 18:01:42.514145 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:01:42.514115 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b1845b8_af82_4fd6_8950_cfccf0c9ea8e.slice/crio-31fb05635e85fec480ae47f4d31be684a860a1dad143e696c06672858d4363ea WatchSource:0}: Error finding container 31fb05635e85fec480ae47f4d31be684a860a1dad143e696c06672858d4363ea: Status 404 returned error can't find the container with id 31fb05635e85fec480ae47f4d31be684a860a1dad143e696c06672858d4363ea Nov 02 18:01:42.554142 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:42.554111 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx" event={"ID":"9b1845b8-af82-4fd6-8950-cfccf0c9ea8e","Type":"ContainerStarted","Data":"31fb05635e85fec480ae47f4d31be684a860a1dad143e696c06672858d4363ea"} Nov 02 18:01:42.789696 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:42.789630 4471 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 02 18:01:42.794486 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:42.794447 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" podUID="8210f11c-5046-4e0b-a3ca-d859918e76c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 02 18:01:42.860036 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:42.859991 4471 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 02 18:01:42.860345 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:42.860284 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" podUID="6ff6c042-8820-4706-bef8-e4cf2732a389" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 02 18:01:43.864068 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:43.854757 4471 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 02 18:01:43.864068 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:43.854805 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" podUID="8210f11c-5046-4e0b-a3ca-d859918e76c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 02 18:01:43.864068 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:43.855269 4471 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 02 18:01:43.864068 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:43.855349 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" podUID="6ff6c042-8820-4706-bef8-e4cf2732a389" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 02 18:01:43.918396 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:43.917096 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-063f48-578558f8d7-r97k5" Nov 02 18:01:43.931252 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:43.931183 4471 generic.go:356] "Generic (PLEG): container finished" podID="1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9" containerID="6eb740b19e330c59d7eca67580fdffb8026b9019b8cbb1a9b365f37b1d264cec" exitCode=0 Nov 02 18:01:43.931418 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:43.931288 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-063f48-578558f8d7-r97k5" event={"ID":"1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9","Type":"ContainerDied","Data":"6eb740b19e330c59d7eca67580fdffb8026b9019b8cbb1a9b365f37b1d264cec"} Nov 02 18:01:43.931418 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:43.931298 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-063f48-578558f8d7-r97k5" Nov 02 18:01:43.931418 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:43.931355 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-063f48-578558f8d7-r97k5" event={"ID":"1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9","Type":"ContainerDied","Data":"d0cc07e363795d3a3136de1ca0cc36dc2e57d323dfb696dae419099ac51183d2"} Nov 02 18:01:43.931418 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:43.931389 4471 scope.go:122] "RemoveContainer" containerID="6eb740b19e330c59d7eca67580fdffb8026b9019b8cbb1a9b365f37b1d264cec" Nov 02 18:01:43.935673 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:43.935649 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx" event={"ID":"9b1845b8-af82-4fd6-8950-cfccf0c9ea8e","Type":"ContainerStarted","Data":"16a29e98549e5d0817ffa7047de72da3177ef7efcad7d57b45579a0d515c51c0"} Nov 02 18:01:43.943671 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:43.943650 4471 scope.go:122] "RemoveContainer" containerID="6eb740b19e330c59d7eca67580fdffb8026b9019b8cbb1a9b365f37b1d264cec" Nov 02 18:01:43.943971 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:01:43.943948 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eb740b19e330c59d7eca67580fdffb8026b9019b8cbb1a9b365f37b1d264cec\": container with ID starting with 6eb740b19e330c59d7eca67580fdffb8026b9019b8cbb1a9b365f37b1d264cec not found: ID does not exist" containerID="6eb740b19e330c59d7eca67580fdffb8026b9019b8cbb1a9b365f37b1d264cec" Nov 02 18:01:43.944063 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:43.943979 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eb740b19e330c59d7eca67580fdffb8026b9019b8cbb1a9b365f37b1d264cec"} err="failed to get container status \"6eb740b19e330c59d7eca67580fdffb8026b9019b8cbb1a9b365f37b1d264cec\": rpc error: code = NotFound desc = could not find container \"6eb740b19e330c59d7eca67580fdffb8026b9019b8cbb1a9b365f37b1d264cec\": container with ID starting with 6eb740b19e330c59d7eca67580fdffb8026b9019b8cbb1a9b365f37b1d264cec not found: ID does not exist" Nov 02 18:01:44.027406 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:44.027355 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx" podStartSLOduration=4.027340292 podStartE2EDuration="4.027340292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:44.024647653 +0000 UTC m=+1921.720003129" watchObservedRunningTime="2026-11-02 18:01:44.027340292 +0000 UTC m=+1921.722695765" Nov 02 18:01:44.106813 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:44.106774 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9-kube-api-access-mwjb6\" (UniqueName: \"kubernetes.io/projected/1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9-kube-api-access-mwjb6\") pod \"1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9\" (UID: \"1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9\") " Nov 02 18:01:44.107985 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:44.107953 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9-plugins-config-volume\") pod \"1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9\" (UID: \"1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9\") " Nov 02 18:01:44.108485 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:44.108455 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9-plugins-config-volume" pod "1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9" (UID: "1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:01:44.111092 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:44.111066 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:01:44.111358 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:44.111332 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9-kube-api-access-mwjb6" pod "1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9" (UID: "1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9"). InnerVolumeSpecName "kube-api-access-mwjb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:44.218717 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:44.218629 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwjb6\" (UniqueName: \"kubernetes.io/projected/1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9-kube-api-access-mwjb6\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:01:44.411201 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:44.411162 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-063f48-578558f8d7-r97k5"] Nov 02 18:01:44.418824 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:44.418795 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-063f48-578558f8d7-r97k5"] Nov 02 18:01:44.446391 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:44.445352 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9" path="/var/lib/kubelet/pods/1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9/volumes" Nov 02 18:01:44.492747 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:01:44.492703 4471 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c55e1d7_ad0e_4491_a301_7d5ccf4cbfa9.slice/crio-d0cc07e363795d3a3136de1ca0cc36dc2e57d323dfb696dae419099ac51183d2\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c55e1d7_ad0e_4491_a301_7d5ccf4cbfa9.slice\": RecentStats: unable to find data in memory cache]" Nov 02 18:01:44.549326 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:44.539548 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx" Nov 02 18:01:44.782386 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:44.778561 4471 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 02 18:01:44.782386 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:44.778614 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" podUID="8210f11c-5046-4e0b-a3ca-d859918e76c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 02 18:01:44.849624 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:44.849587 4471 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 02 18:01:44.849832 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:44.849652 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" podUID="6ff6c042-8820-4706-bef8-e4cf2732a389" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 02 18:01:45.783771 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:45.783738 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:01:45.784260 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:45.784232 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:01:45.784927 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:45.784904 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:01:45.855233 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:45.855201 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:01:45.855565 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:45.855544 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:01:45.856214 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:45.856193 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:01:49.444441 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:49.444405 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" Nov 02 18:01:49.457487 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:49.457453 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" Nov 02 18:01:50.002778 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:50.002749 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" Nov 02 18:01:51.539071 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:51.539037 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx" Nov 02 18:01:51.555968 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:51.555929 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx" Nov 02 18:01:52.019024 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:52.018997 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx" Nov 02 18:01:52.447479 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:52.447370 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn"] Nov 02 18:01:52.448598 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:52.448572 4471 cpu_manager.go:414] "containerMap: removing container" podUID="1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9" containerName="payload-pre-processing" Nov 02 18:01:52.448598 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:52.448595 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9" containerName="payload-pre-processing" Nov 02 18:01:52.448837 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:52.448809 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c55e1d7-ad0e-4491-a301-7d5ccf4cbfa9" containerName="payload-pre-processing" Nov 02 18:01:52.454937 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:52.454917 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn" Nov 02 18:01:52.465103 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:52.465077 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn"] Nov 02 18:01:52.483634 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:52.483605 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-872354\"/\"default-dockercfg-fcbkq\"" Nov 02 18:01:52.483764 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:52.483686 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-872354\"/\"kube-root-ca.crt\"" Nov 02 18:01:52.483764 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:52.483693 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-872354\"/\"test-model-872354-kserve-self-signed-certs\"" Nov 02 18:01:52.539164 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:52.539125 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfvbg\" (UniqueName: \"kubernetes.io/projected/24485a36-8054-44f1-a2ff-f8471e726249-kube-api-access-jfvbg\") pod \"test-model-872354-kserve-8f6cd8fd9-xwvbn\" (UID: \"24485a36-8054-44f1-a2ff-f8471e726249\") " pod="ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn" Nov 02 18:01:52.539604 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:52.539174 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24485a36-8054-44f1-a2ff-f8471e726249-dshm\") pod \"test-model-872354-kserve-8f6cd8fd9-xwvbn\" (UID: \"24485a36-8054-44f1-a2ff-f8471e726249\") " pod="ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn" Nov 02 18:01:52.539604 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:52.539460 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24485a36-8054-44f1-a2ff-f8471e726249-model-cache\") pod \"test-model-872354-kserve-8f6cd8fd9-xwvbn\" (UID: \"24485a36-8054-44f1-a2ff-f8471e726249\") " pod="ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn" Nov 02 18:01:52.539604 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:52.539490 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24485a36-8054-44f1-a2ff-f8471e726249-tls-certs\") pod \"test-model-872354-kserve-8f6cd8fd9-xwvbn\" (UID: \"24485a36-8054-44f1-a2ff-f8471e726249\") " pod="ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn" Nov 02 18:01:52.539604 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:52.539590 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24485a36-8054-44f1-a2ff-f8471e726249-home\") pod \"test-model-872354-kserve-8f6cd8fd9-xwvbn\" (UID: \"24485a36-8054-44f1-a2ff-f8471e726249\") " pod="ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn" Nov 02 18:01:52.539956 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:52.539711 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24485a36-8054-44f1-a2ff-f8471e726249-tmp-dir\") pod \"test-model-872354-kserve-8f6cd8fd9-xwvbn\" (UID: \"24485a36-8054-44f1-a2ff-f8471e726249\") " pod="ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn" Nov 02 18:01:52.571351 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:52.571322 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-872354\"/\"openshift-service-ca.crt\"" Nov 02 18:01:52.641530 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:52.641497 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfvbg\" (UniqueName: \"kubernetes.io/projected/24485a36-8054-44f1-a2ff-f8471e726249-kube-api-access-jfvbg\") pod \"test-model-872354-kserve-8f6cd8fd9-xwvbn\" (UID: \"24485a36-8054-44f1-a2ff-f8471e726249\") " pod="ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn" Nov 02 18:01:52.641701 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:52.641540 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24485a36-8054-44f1-a2ff-f8471e726249-dshm\") pod \"test-model-872354-kserve-8f6cd8fd9-xwvbn\" (UID: \"24485a36-8054-44f1-a2ff-f8471e726249\") " pod="ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn" Nov 02 18:01:52.641745 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:52.641703 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24485a36-8054-44f1-a2ff-f8471e726249-model-cache\") pod \"test-model-872354-kserve-8f6cd8fd9-xwvbn\" (UID: \"24485a36-8054-44f1-a2ff-f8471e726249\") " pod="ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn" Nov 02 18:01:52.641745 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:52.641722 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24485a36-8054-44f1-a2ff-f8471e726249-tls-certs\") pod \"test-model-872354-kserve-8f6cd8fd9-xwvbn\" (UID: \"24485a36-8054-44f1-a2ff-f8471e726249\") " pod="ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn" Nov 02 18:01:52.641814 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:52.641758 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24485a36-8054-44f1-a2ff-f8471e726249-home\") pod \"test-model-872354-kserve-8f6cd8fd9-xwvbn\" (UID: \"24485a36-8054-44f1-a2ff-f8471e726249\") " pod="ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn" Nov 02 18:01:52.641859 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:52.641836 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24485a36-8054-44f1-a2ff-f8471e726249-tmp-dir\") pod \"test-model-872354-kserve-8f6cd8fd9-xwvbn\" (UID: \"24485a36-8054-44f1-a2ff-f8471e726249\") " pod="ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn" Nov 02 18:01:52.642142 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:52.642112 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24485a36-8054-44f1-a2ff-f8471e726249-model-cache\") pod \"test-model-872354-kserve-8f6cd8fd9-xwvbn\" (UID: \"24485a36-8054-44f1-a2ff-f8471e726249\") " pod="ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn" Nov 02 18:01:52.642142 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:52.642136 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24485a36-8054-44f1-a2ff-f8471e726249-home\") pod \"test-model-872354-kserve-8f6cd8fd9-xwvbn\" (UID: \"24485a36-8054-44f1-a2ff-f8471e726249\") " pod="ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn" Nov 02 18:01:52.642290 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:52.642173 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24485a36-8054-44f1-a2ff-f8471e726249-tmp-dir\") pod \"test-model-872354-kserve-8f6cd8fd9-xwvbn\" (UID: \"24485a36-8054-44f1-a2ff-f8471e726249\") " pod="ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn" Nov 02 18:01:52.644025 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:52.644008 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24485a36-8054-44f1-a2ff-f8471e726249-dshm\") pod \"test-model-872354-kserve-8f6cd8fd9-xwvbn\" (UID: \"24485a36-8054-44f1-a2ff-f8471e726249\") " pod="ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn" Nov 02 18:01:52.644145 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:52.644125 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24485a36-8054-44f1-a2ff-f8471e726249-tls-certs\") pod \"test-model-872354-kserve-8f6cd8fd9-xwvbn\" (UID: \"24485a36-8054-44f1-a2ff-f8471e726249\") " pod="ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn" Nov 02 18:01:52.666075 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:52.666046 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfvbg\" (UniqueName: \"kubernetes.io/projected/24485a36-8054-44f1-a2ff-f8471e726249-kube-api-access-jfvbg\") pod \"test-model-872354-kserve-8f6cd8fd9-xwvbn\" (UID: \"24485a36-8054-44f1-a2ff-f8471e726249\") " pod="ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn" Nov 02 18:01:52.775711 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:52.775670 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn" Nov 02 18:01:53.293287 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:01:53.293248 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24485a36_8054_44f1_a2ff_f8471e726249.slice/crio-d4850b1964293d5356ba6d6378fb5632c7887a71cc24200a2a29856555d834d3 WatchSource:0}: Error finding container d4850b1964293d5356ba6d6378fb5632c7887a71cc24200a2a29856555d834d3: Status 404 returned error can't find the container with id d4850b1964293d5356ba6d6378fb5632c7887a71cc24200a2a29856555d834d3 Nov 02 18:01:53.294293 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:53.294265 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn"] Nov 02 18:01:53.879169 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:53.879132 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr"] Nov 02 18:01:53.884377 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:53.884339 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:01:53.910445 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:53.910412 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr"] Nov 02 18:01:53.923842 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:53.923620 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-fbbbf8-openshift-default-dockercfg-b9pj2\"" Nov 02 18:01:54.035225 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.035190 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ec749a-738f-4509-a7e5-be6eb97f4777-istio-podinfo\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:01:54.035418 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.035301 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-workload-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:01:54.035523 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.035501 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-istio-envoy\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:01:54.035580 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.035534 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-workload-certs\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:01:54.035580 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.035554 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-istio-data\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:01:54.035779 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.035758 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ec749a-738f-4509-a7e5-be6eb97f4777-istiod-ca-cert\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:01:54.035851 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.035820 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-credential-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:01:54.035988 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.035968 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8xkw\" (UniqueName: \"kubernetes.io/projected/74ec749a-738f-4509-a7e5-be6eb97f4777-kube-api-access-k8xkw\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:01:54.036085 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.036071 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ec749a-738f-4509-a7e5-be6eb97f4777-istio-token\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:01:54.038632 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.038604 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn" event={"ID":"24485a36-8054-44f1-a2ff-f8471e726249","Type":"ContainerStarted","Data":"e4320f149008676e4c5a3c9e9cc691584fe8d302ca40415bfb596b6796d5231f"} Nov 02 18:01:54.038745 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.038652 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn" event={"ID":"24485a36-8054-44f1-a2ff-f8471e726249","Type":"ContainerStarted","Data":"d4850b1964293d5356ba6d6378fb5632c7887a71cc24200a2a29856555d834d3"} Nov 02 18:01:54.086631 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.086583 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn" podStartSLOduration=2.086568004 podStartE2EDuration="2.086568004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:54.081448198 +0000 UTC m=+1931.776803700" watchObservedRunningTime="2026-11-02 18:01:54.086568004 +0000 UTC m=+1931.781923499" Nov 02 18:01:54.137493 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.137420 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ec749a-738f-4509-a7e5-be6eb97f4777-istio-podinfo\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:01:54.137493 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.137465 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-workload-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:01:54.137666 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.137541 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-istio-envoy\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:01:54.137666 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.137559 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-workload-certs\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:01:54.137666 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.137575 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-istio-data\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:01:54.137820 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.137671 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ec749a-738f-4509-a7e5-be6eb97f4777-istiod-ca-cert\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:01:54.137820 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.137713 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-credential-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:01:54.137932 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.137816 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8xkw\" (UniqueName: \"kubernetes.io/projected/74ec749a-738f-4509-a7e5-be6eb97f4777-kube-api-access-k8xkw\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:01:54.137932 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.137909 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ec749a-738f-4509-a7e5-be6eb97f4777-istio-token\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:01:54.138036 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.137991 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-workload-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:01:54.138036 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.138024 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-workload-certs\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:01:54.138384 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.138283 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-istio-data\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:01:54.138547 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.138526 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ec749a-738f-4509-a7e5-be6eb97f4777-istiod-ca-cert\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:01:54.138590 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.138555 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-credential-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:01:54.139846 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.139825 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-istio-envoy\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:01:54.140088 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.140070 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ec749a-738f-4509-a7e5-be6eb97f4777-istio-podinfo\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:01:54.162351 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.162299 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ec749a-738f-4509-a7e5-be6eb97f4777-istio-token\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:01:54.162578 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.162557 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8xkw\" (UniqueName: \"kubernetes.io/projected/74ec749a-738f-4509-a7e5-be6eb97f4777-kube-api-access-k8xkw\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:01:54.235394 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.228029 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:01:54.610062 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:01:54.610031 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74ec749a_738f_4509_a7e5_be6eb97f4777.slice/crio-3dc1ae34312fd0b8c8f712471644f41672cf659b5ca92e324417c61dcad0d449 WatchSource:0}: Error finding container 3dc1ae34312fd0b8c8f712471644f41672cf659b5ca92e324417c61dcad0d449: Status 404 returned error can't find the container with id 3dc1ae34312fd0b8c8f712471644f41672cf659b5ca92e324417c61dcad0d449 Nov 02 18:01:54.613629 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.613593 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:01:54.613710 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.613664 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:01:54.613710 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.613695 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:01:54.615411 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.615389 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr"] Nov 02 18:01:54.784357 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:54.776517 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn" Nov 02 18:01:55.049607 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:55.049570 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" event={"ID":"74ec749a-738f-4509-a7e5-be6eb97f4777","Type":"ContainerStarted","Data":"3a8171f7433846b9317402622b74a5b56ead44961ed13c1ef3f5bef281544230"} Nov 02 18:01:55.050014 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:55.049651 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" event={"ID":"74ec749a-738f-4509-a7e5-be6eb97f4777","Type":"ContainerStarted","Data":"3dc1ae34312fd0b8c8f712471644f41672cf659b5ca92e324417c61dcad0d449"} Nov 02 18:01:55.110478 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:55.110424 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" podStartSLOduration=2.110409588 podStartE2EDuration="2.110409588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:55.109886028 +0000 UTC m=+1932.805241504" watchObservedRunningTime="2026-11-02 18:01:55.110409588 +0000 UTC m=+1932.805765061" Nov 02 18:01:55.236006 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:55.235973 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:01:55.237255 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:55.237227 4471 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Nov 02 18:01:55.237424 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:55.237279 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" podUID="74ec749a-738f-4509-a7e5-be6eb97f4777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Nov 02 18:01:56.236641 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:56.236610 4471 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Nov 02 18:01:56.237039 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:56.236660 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" podUID="74ec749a-738f-4509-a7e5-be6eb97f4777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Nov 02 18:01:57.236178 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:57.236147 4471 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Nov 02 18:01:57.236382 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:57.236200 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" podUID="74ec749a-738f-4509-a7e5-be6eb97f4777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Nov 02 18:01:58.236245 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:58.236204 4471 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Nov 02 18:01:58.236655 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:58.236271 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" podUID="74ec749a-738f-4509-a7e5-be6eb97f4777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Nov 02 18:01:59.236071 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:59.236034 4471 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Nov 02 18:01:59.236300 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:01:59.236086 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" podUID="74ec749a-738f-4509-a7e5-be6eb97f4777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Nov 02 18:02:00.236014 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:00.235980 4471 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Nov 02 18:02:00.236208 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:00.236034 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" podUID="74ec749a-738f-4509-a7e5-be6eb97f4777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Nov 02 18:02:01.241277 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:01.241246 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:02:01.241789 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:01.241729 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:02:01.242490 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:01.242469 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:02:02.776932 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:02.776896 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn" Nov 02 18:02:02.789344 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:02.789299 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn" Nov 02 18:02:03.174666 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:03.174583 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn" Nov 02 18:02:09.203955 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:02:09.203918 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:02:17.135791 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:17.135752 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw"] Nov 02 18:02:17.143434 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:17.143410 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw" Nov 02 18:02:17.162057 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:17.162029 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw"] Nov 02 18:02:17.172988 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:17.172962 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fbbbf8\"/\"openshift-service-ca.crt\"" Nov 02 18:02:17.173212 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:17.173083 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fbbbf8\"/\"kube-root-ca.crt\"" Nov 02 18:02:17.173387 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:17.173127 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fbbbf8\"/\"default-dockercfg-lq5j7\"" Nov 02 18:02:17.173387 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:17.173139 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fbbbf8\"/\"test-model-fbbbf8-kserve-self-signed-certs\"" Nov 02 18:02:17.318804 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:17.318768 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67kmv\" (UniqueName: \"kubernetes.io/projected/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-kube-api-access-67kmv\") pod \"test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw\" (UID: \"af93e51c-6948-43fd-b8a3-c7fd0c730dc6\") " pod="ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw" Nov 02 18:02:17.318979 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:17.318837 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-home\") pod \"test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw\" (UID: \"af93e51c-6948-43fd-b8a3-c7fd0c730dc6\") " pod="ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw" Nov 02 18:02:17.319027 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:17.318979 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-dshm\") pod \"test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw\" (UID: \"af93e51c-6948-43fd-b8a3-c7fd0c730dc6\") " pod="ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw" Nov 02 18:02:17.319068 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:17.319044 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-model-cache\") pod \"test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw\" (UID: \"af93e51c-6948-43fd-b8a3-c7fd0c730dc6\") " pod="ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw" Nov 02 18:02:17.319220 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:17.319203 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-tls-certs\") pod \"test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw\" (UID: \"af93e51c-6948-43fd-b8a3-c7fd0c730dc6\") " pod="ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw" Nov 02 18:02:17.319289 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:17.319276 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-tmp-dir\") pod \"test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw\" (UID: \"af93e51c-6948-43fd-b8a3-c7fd0c730dc6\") " pod="ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw" Nov 02 18:02:17.430101 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:17.430010 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67kmv\" (UniqueName: \"kubernetes.io/projected/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-kube-api-access-67kmv\") pod \"test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw\" (UID: \"af93e51c-6948-43fd-b8a3-c7fd0c730dc6\") " pod="ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw" Nov 02 18:02:17.430508 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:17.430482 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-home\") pod \"test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw\" (UID: \"af93e51c-6948-43fd-b8a3-c7fd0c730dc6\") " pod="ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw" Nov 02 18:02:17.430756 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:17.430741 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-dshm\") pod \"test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw\" (UID: \"af93e51c-6948-43fd-b8a3-c7fd0c730dc6\") " pod="ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw" Nov 02 18:02:17.430928 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:17.430915 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-model-cache\") pod \"test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw\" (UID: \"af93e51c-6948-43fd-b8a3-c7fd0c730dc6\") " pod="ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw" Nov 02 18:02:17.431119 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:17.431103 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-tls-certs\") pod \"test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw\" (UID: \"af93e51c-6948-43fd-b8a3-c7fd0c730dc6\") " pod="ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw" Nov 02 18:02:17.431280 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:17.431266 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-tmp-dir\") pod \"test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw\" (UID: \"af93e51c-6948-43fd-b8a3-c7fd0c730dc6\") " pod="ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw" Nov 02 18:02:17.431933 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:17.431911 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-tmp-dir\") pod \"test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw\" (UID: \"af93e51c-6948-43fd-b8a3-c7fd0c730dc6\") " pod="ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw" Nov 02 18:02:17.432581 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:17.432559 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-home\") pod \"test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw\" (UID: \"af93e51c-6948-43fd-b8a3-c7fd0c730dc6\") " pod="ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw" Nov 02 18:02:17.433570 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:17.432897 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-model-cache\") pod \"test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw\" (UID: \"af93e51c-6948-43fd-b8a3-c7fd0c730dc6\") " pod="ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw" Nov 02 18:02:17.454334 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:17.438191 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-tls-certs\") pod \"test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw\" (UID: \"af93e51c-6948-43fd-b8a3-c7fd0c730dc6\") " pod="ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw" Nov 02 18:02:17.464327 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:17.454937 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-dshm\") pod \"test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw\" (UID: \"af93e51c-6948-43fd-b8a3-c7fd0c730dc6\") " pod="ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw" Nov 02 18:02:17.497328 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:17.479617 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67kmv\" (UniqueName: \"kubernetes.io/projected/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-kube-api-access-67kmv\") pod \"test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw\" (UID: \"af93e51c-6948-43fd-b8a3-c7fd0c730dc6\") " pod="ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw" Nov 02 18:02:17.758444 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:17.758344 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw" Nov 02 18:02:18.066407 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:02:18.066374 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf93e51c_6948_43fd_b8a3_c7fd0c730dc6.slice/crio-e0171b7f1bdbb183059fd697b101a93b3d0f34cdd49106643d8c2c161e044663 WatchSource:0}: Error finding container e0171b7f1bdbb183059fd697b101a93b3d0f34cdd49106643d8c2c161e044663: Status 404 returned error can't find the container with id e0171b7f1bdbb183059fd697b101a93b3d0f34cdd49106643d8c2c161e044663 Nov 02 18:02:18.069299 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:18.069273 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw"] Nov 02 18:02:18.305404 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:18.305362 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw" event={"ID":"af93e51c-6948-43fd-b8a3-c7fd0c730dc6","Type":"ContainerStarted","Data":"88ec3ce0aaef54c4c198e3fdef0834ecceb0d07e86087305d1ec2d728b02e02e"} Nov 02 18:02:18.305404 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:18.305407 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw" event={"ID":"af93e51c-6948-43fd-b8a3-c7fd0c730dc6","Type":"ContainerStarted","Data":"e0171b7f1bdbb183059fd697b101a93b3d0f34cdd49106643d8c2c161e044663"} Nov 02 18:02:18.355325 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:18.355209 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw" podStartSLOduration=1.3551935419999999 podStartE2EDuration="1.355193542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:02:18.349230882 +0000 UTC m=+1956.044586375" watchObservedRunningTime="2026-11-02 18:02:18.355193542 +0000 UTC m=+1956.050549029" Nov 02 18:02:18.768234 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:18.764430 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw" Nov 02 18:02:27.758771 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:27.758738 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw" Nov 02 18:02:27.771037 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:27.771012 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw" Nov 02 18:02:28.450286 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:02:28.450255 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw" Nov 02 18:02:39.187054 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:02:39.187016 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:03:08.527754 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:08.527719 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:03:09.187729 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:03:09.187688 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:03:11.363675 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:11.363627 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:11.382453 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:11.382403 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" podUID="c3e6eb3c-0496-44c8-a45f-9b4c662756b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.155:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:11.540767 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:11.540717 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:11.628988 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:11.628850 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:11.679401 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:11.673684 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 02 18:03:11.708375 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:11.708325 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:11.709062 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:11.709024 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" podUID="b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.153:8000/ready\": context deadline exceeded" Nov 02 18:03:11.709538 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:11.709514 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 18:03:11.841382 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:11.832410 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:12.006146 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:12.005451 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" podUID="fad2db08-e940-4823-99ee-9b3fb1346140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.180:8000/ready\": context deadline exceeded" Nov 02 18:03:12.028804 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:12.028747 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx" podUID="9b1845b8-af82-4fd6-8950-cfccf0c9ea8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.181:8000/ready\": context deadline exceeded" Nov 02 18:03:12.185397 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:12.185352 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn" podUID="24485a36-8054-44f1-a2ff-f8471e726249" containerName="main" probeResult="failure" output="Get \"https://10.133.0.182:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:12.443329 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:12.422024 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:12.498233 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:12.447433 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw" podUID="af93e51c-6948-43fd-b8a3-c7fd0c730dc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.184:8000/ready\": context deadline exceeded" Nov 02 18:03:12.498780 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:12.447490 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" podUID="c3e6eb3c-0496-44c8-a45f-9b4c662756b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.155:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:12.498982 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:12.497936 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm" podUID="a4d08194-c737-4a26-9198-ecfd8e6554f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:12.565380 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:12.550330 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" podUID="5c5408ad-4b7b-47c0-854a-6e6c3a723302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:12.565697 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:12.561192 4471 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx" podUID="9b1845b8-af82-4fd6-8950-cfccf0c9ea8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.181:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:12.593332 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:12.590022 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" podUID="33601c55-3e8f-41d4-a13e-1099a26334f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.151:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:12.669009 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:12.668958 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 18:03:12.737429 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:12.737290 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:12.738159 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:12.738127 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 02 18:03:12.742556 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:12.742525 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" podUID="b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.153:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:12.742744 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:12.742723 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 18:03:12.837273 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:12.836269 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:12.849329 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:12.839603 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 18:03:12.849678 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:12.849654 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" Nov 02 18:03:12.849878 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:12.849865 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 18:03:12.850003 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:12.849993 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 18:03:12.850126 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:12.850116 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 18:03:12.850243 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:12.850233 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 18:03:12.850381 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:12.850371 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" Nov 02 18:03:12.850546 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:12.850535 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 18:03:12.850670 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:12.850660 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 18:03:12.999604 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:12.999496 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" Nov 02 18:03:12.999604 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:12.999587 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 18:03:13.010946 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:13.008693 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" podUID="fad2db08-e940-4823-99ee-9b3fb1346140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.180:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:13.010946 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:13.009176 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" Nov 02 18:03:13.014545 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:13.014296 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 18:03:13.014545 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:13.014366 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 18:03:13.019643 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:13.016137 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h" Nov 02 18:03:13.019643 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:13.016888 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 18:03:13.019643 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:13.017359 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 18:03:13.022734 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:13.022711 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" Nov 02 18:03:13.042610 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:13.042451 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" Nov 02 18:03:13.050744 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:13.050712 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 18:03:20.236367 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:20.236328 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l"] Nov 02 18:03:20.239105 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:20.236546 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" podUID="49caa894-b067-4d85-8141-741e72c8f041" containerName="istio-proxy" containerID="cri-o://722539c79b3ec3867dbaaf17bb6298974482efc261c1181334e204b6499f5779" gracePeriod=30 Nov 02 18:03:21.124697 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:21.124660 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv"] Nov 02 18:03:21.124938 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:21.124909 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" podUID="9c572702-1be9-42ec-a4b7-243fe8bfd9c8" containerName="istio-proxy" containerID="cri-o://d0a03b0dd74fe6c8c960e59fa859d7379f67979ef8a98fa11419476cfa87702d" gracePeriod=30 Nov 02 18:03:24.570574 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:24.570535 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:03:24.570965 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:24.570603 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" podUID="9c572702-1be9-42ec-a4b7-243fe8bfd9c8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:03:25.546487 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:25.546459 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 18:03:25.629870 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:25.629840 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49caa894-b067-4d85-8141-741e72c8f041-kube-api-access-vvk57\" (UniqueName: \"kubernetes.io/projected/49caa894-b067-4d85-8141-741e72c8f041-kube-api-access-vvk57\") pod \"49caa894-b067-4d85-8141-741e72c8f041\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " Nov 02 18:03:25.630401 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:25.629921 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-istio-envoy\") pod \"49caa894-b067-4d85-8141-741e72c8f041\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " Nov 02 18:03:25.630564 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:25.630540 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49caa894-b067-4d85-8141-741e72c8f041-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49caa894-b067-4d85-8141-741e72c8f041-istiod-ca-cert\") pod \"49caa894-b067-4d85-8141-741e72c8f041\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " Nov 02 18:03:25.630903 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:25.630879 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-workload-certs\") pod \"49caa894-b067-4d85-8141-741e72c8f041\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " Nov 02 18:03:25.631351 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:25.631287 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49caa894-b067-4d85-8141-741e72c8f041-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49caa894-b067-4d85-8141-741e72c8f041-istio-podinfo\") pod \"49caa894-b067-4d85-8141-741e72c8f041\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " Nov 02 18:03:25.632228 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:25.632207 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49caa894-b067-4d85-8141-741e72c8f041-istio-token\" (UniqueName: \"kubernetes.io/projected/49caa894-b067-4d85-8141-741e72c8f041-istio-token\") pod \"49caa894-b067-4d85-8141-741e72c8f041\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " Nov 02 18:03:25.632549 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:25.632528 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-credential-socket\") pod \"49caa894-b067-4d85-8141-741e72c8f041\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " Nov 02 18:03:25.632745 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:25.632724 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-workload-socket\") pod \"49caa894-b067-4d85-8141-741e72c8f041\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " Nov 02 18:03:25.632895 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:25.632862 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49caa894-b067-4d85-8141-741e72c8f041-kube-api-access-vvk57" pod "49caa894-b067-4d85-8141-741e72c8f041" (UID: "49caa894-b067-4d85-8141-741e72c8f041"). InnerVolumeSpecName "kube-api-access-vvk57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:03:25.633021 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:25.633001 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-istio-data\") pod \"49caa894-b067-4d85-8141-741e72c8f041\" (UID: \"49caa894-b067-4d85-8141-741e72c8f041\") " Nov 02 18:03:25.633531 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:25.633506 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-istio-envoy" pod "49caa894-b067-4d85-8141-741e72c8f041" (UID: "49caa894-b067-4d85-8141-741e72c8f041"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:03:25.633837 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:25.633815 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49caa894-b067-4d85-8141-741e72c8f041-istiod-ca-cert" pod "49caa894-b067-4d85-8141-741e72c8f041" (UID: "49caa894-b067-4d85-8141-741e72c8f041"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:03:25.634072 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:25.634052 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-credential-socket" pod "49caa894-b067-4d85-8141-741e72c8f041" (UID: "49caa894-b067-4d85-8141-741e72c8f041"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:03:25.634637 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:25.634615 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvk57\" (UniqueName: \"kubernetes.io/projected/49caa894-b067-4d85-8141-741e72c8f041-kube-api-access-vvk57\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:03:25.634790 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:25.634756 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:03:25.634898 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:25.634886 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49caa894-b067-4d85-8141-741e72c8f041-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:03:25.634997 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:25.634984 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:03:25.635284 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:25.635256 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49caa894-b067-4d85-8141-741e72c8f041-istio-podinfo" pod "49caa894-b067-4d85-8141-741e72c8f041" (UID: "49caa894-b067-4d85-8141-741e72c8f041"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:03:25.635421 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:25.635274 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-workload-socket" pod "49caa894-b067-4d85-8141-741e72c8f041" (UID: "49caa894-b067-4d85-8141-741e72c8f041"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:03:25.635546 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:25.635524 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-workload-certs" pod "49caa894-b067-4d85-8141-741e72c8f041" (UID: "49caa894-b067-4d85-8141-741e72c8f041"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:03:25.635875 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:25.635853 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-istio-data" pod "49caa894-b067-4d85-8141-741e72c8f041" (UID: "49caa894-b067-4d85-8141-741e72c8f041"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:03:25.636292 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:25.636259 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49caa894-b067-4d85-8141-741e72c8f041-istio-token" pod "49caa894-b067-4d85-8141-741e72c8f041" (UID: "49caa894-b067-4d85-8141-741e72c8f041"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:03:25.736883 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:25.736852 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:03:25.736883 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:25.736882 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49caa894-b067-4d85-8141-741e72c8f041-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:03:25.736883 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:25.736891 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49caa894-b067-4d85-8141-741e72c8f041-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:03:25.737121 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:25.736900 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:03:25.737121 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:25.736907 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49caa894-b067-4d85-8141-741e72c8f041-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:03:26.365843 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.365804 4471 generic.go:356] "Generic (PLEG): container finished" podID="9c572702-1be9-42ec-a4b7-243fe8bfd9c8" containerID="d0a03b0dd74fe6c8c960e59fa859d7379f67979ef8a98fa11419476cfa87702d" exitCode=0 Nov 02 18:03:26.366087 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.365878 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" event={"ID":"9c572702-1be9-42ec-a4b7-243fe8bfd9c8","Type":"ContainerDied","Data":"d0a03b0dd74fe6c8c960e59fa859d7379f67979ef8a98fa11419476cfa87702d"} Nov 02 18:03:26.368870 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.368840 4471 generic.go:356] "Generic (PLEG): container finished" podID="49caa894-b067-4d85-8141-741e72c8f041" containerID="722539c79b3ec3867dbaaf17bb6298974482efc261c1181334e204b6499f5779" exitCode=0 Nov 02 18:03:26.369009 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.368897 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" event={"ID":"49caa894-b067-4d85-8141-741e72c8f041","Type":"ContainerDied","Data":"722539c79b3ec3867dbaaf17bb6298974482efc261c1181334e204b6499f5779"} Nov 02 18:03:26.369009 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.368942 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" event={"ID":"49caa894-b067-4d85-8141-741e72c8f041","Type":"ContainerDied","Data":"9b6fc0af8f78729ba08112f5f23719b1d73166959077a306a87cca2b1969547b"} Nov 02 18:03:26.369009 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.368974 4471 scope.go:122] "RemoveContainer" containerID="722539c79b3ec3867dbaaf17bb6298974482efc261c1181334e204b6499f5779" Nov 02 18:03:26.369189 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.369147 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l" Nov 02 18:03:26.388373 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.388350 4471 scope.go:122] "RemoveContainer" containerID="1d452628e1a1a514cdc498bf3e85994af2e768ec805be2df240146fedfe81cbe" Nov 02 18:03:26.431462 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.431442 4471 scope.go:122] "RemoveContainer" containerID="722539c79b3ec3867dbaaf17bb6298974482efc261c1181334e204b6499f5779" Nov 02 18:03:26.431739 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:03:26.431714 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"722539c79b3ec3867dbaaf17bb6298974482efc261c1181334e204b6499f5779\": container with ID starting with 722539c79b3ec3867dbaaf17bb6298974482efc261c1181334e204b6499f5779 not found: ID does not exist" containerID="722539c79b3ec3867dbaaf17bb6298974482efc261c1181334e204b6499f5779" Nov 02 18:03:26.431786 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.431752 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"722539c79b3ec3867dbaaf17bb6298974482efc261c1181334e204b6499f5779"} err="failed to get container status \"722539c79b3ec3867dbaaf17bb6298974482efc261c1181334e204b6499f5779\": rpc error: code = NotFound desc = could not find container \"722539c79b3ec3867dbaaf17bb6298974482efc261c1181334e204b6499f5779\": container with ID starting with 722539c79b3ec3867dbaaf17bb6298974482efc261c1181334e204b6499f5779 not found: ID does not exist" Nov 02 18:03:26.431786 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.431771 4471 scope.go:122] "RemoveContainer" containerID="1d452628e1a1a514cdc498bf3e85994af2e768ec805be2df240146fedfe81cbe" Nov 02 18:03:26.431999 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:03:26.431983 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d452628e1a1a514cdc498bf3e85994af2e768ec805be2df240146fedfe81cbe\": container with ID starting with 1d452628e1a1a514cdc498bf3e85994af2e768ec805be2df240146fedfe81cbe not found: ID does not exist" containerID="1d452628e1a1a514cdc498bf3e85994af2e768ec805be2df240146fedfe81cbe" Nov 02 18:03:26.432046 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.432005 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d452628e1a1a514cdc498bf3e85994af2e768ec805be2df240146fedfe81cbe"} err="failed to get container status \"1d452628e1a1a514cdc498bf3e85994af2e768ec805be2df240146fedfe81cbe\": rpc error: code = NotFound desc = could not find container \"1d452628e1a1a514cdc498bf3e85994af2e768ec805be2df240146fedfe81cbe\": container with ID starting with 1d452628e1a1a514cdc498bf3e85994af2e768ec805be2df240146fedfe81cbe not found: ID does not exist" Nov 02 18:03:26.433150 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.433130 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 18:03:26.480516 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.480489 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l"] Nov 02 18:03:26.496355 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.496325 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-4vc6l"] Nov 02 18:03:26.551164 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.551134 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-workload-socket\") pod \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " Nov 02 18:03:26.551372 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.551205 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istio-podinfo\") pod \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " Nov 02 18:03:26.551372 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.551243 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istiod-ca-cert\") pod \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " Nov 02 18:03:26.551470 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.551437 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istio-token\" (UniqueName: \"kubernetes.io/projected/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istio-token\") pod \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " Nov 02 18:03:26.551520 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.551482 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-workload-socket" pod "9c572702-1be9-42ec-a4b7-243fe8bfd9c8" (UID: "9c572702-1be9-42ec-a4b7-243fe8bfd9c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:03:26.551520 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.551495 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istio-data\") pod \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " Nov 02 18:03:26.551654 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.551553 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-credential-socket\") pod \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " Nov 02 18:03:26.551715 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.551660 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istiod-ca-cert" pod "9c572702-1be9-42ec-a4b7-243fe8bfd9c8" (UID: "9c572702-1be9-42ec-a4b7-243fe8bfd9c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:03:26.551715 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.551679 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-workload-certs\") pod \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " Nov 02 18:03:26.551826 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.551767 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istio-data" pod "9c572702-1be9-42ec-a4b7-243fe8bfd9c8" (UID: "9c572702-1be9-42ec-a4b7-243fe8bfd9c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:03:26.551826 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.551776 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istio-envoy\") pod \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " Nov 02 18:03:26.551955 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.551859 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-kube-api-access-rlqvs\" (UniqueName: \"kubernetes.io/projected/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-kube-api-access-rlqvs\") pod \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\" (UID: \"9c572702-1be9-42ec-a4b7-243fe8bfd9c8\") " Nov 02 18:03:26.551955 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.551867 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-credential-socket" pod "9c572702-1be9-42ec-a4b7-243fe8bfd9c8" (UID: "9c572702-1be9-42ec-a4b7-243fe8bfd9c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:03:26.552064 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.551952 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-workload-certs" pod "9c572702-1be9-42ec-a4b7-243fe8bfd9c8" (UID: "9c572702-1be9-42ec-a4b7-243fe8bfd9c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:03:26.553128 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.553093 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:03:26.553128 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.553120 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:03:26.553342 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.553136 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:03:26.553342 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.553150 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:03:26.553342 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.553168 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:03:26.553780 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.553753 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istio-podinfo" pod "9c572702-1be9-42ec-a4b7-243fe8bfd9c8" (UID: "9c572702-1be9-42ec-a4b7-243fe8bfd9c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:03:26.553887 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.553852 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istio-token" pod "9c572702-1be9-42ec-a4b7-243fe8bfd9c8" (UID: "9c572702-1be9-42ec-a4b7-243fe8bfd9c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:03:26.554001 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.553984 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istio-envoy" pod "9c572702-1be9-42ec-a4b7-243fe8bfd9c8" (UID: "9c572702-1be9-42ec-a4b7-243fe8bfd9c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:03:26.554218 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.554197 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-kube-api-access-rlqvs" pod "9c572702-1be9-42ec-a4b7-243fe8bfd9c8" (UID: "9c572702-1be9-42ec-a4b7-243fe8bfd9c8"). InnerVolumeSpecName "kube-api-access-rlqvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:03:26.654833 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.654800 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlqvs\" (UniqueName: \"kubernetes.io/projected/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-kube-api-access-rlqvs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:03:26.654833 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.654832 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:03:26.655246 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.654842 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:03:26.655246 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:26.654850 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c572702-1be9-42ec-a4b7-243fe8bfd9c8-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:03:27.379946 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:27.379913 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" Nov 02 18:03:27.380125 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:27.379947 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv" event={"ID":"9c572702-1be9-42ec-a4b7-243fe8bfd9c8","Type":"ContainerDied","Data":"1d5acf3eba4dd03b9b7c7f9781dfe656bae31b3e8f1694e52946ce6b84e0c6e0"} Nov 02 18:03:27.380125 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:27.380012 4471 scope.go:122] "RemoveContainer" containerID="d0a03b0dd74fe6c8c960e59fa859d7379f67979ef8a98fa11419476cfa87702d" Nov 02 18:03:27.439847 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:27.439759 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv"] Nov 02 18:03:27.448215 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:27.448188 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bbgrv"] Nov 02 18:03:28.444638 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:28.444600 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49caa894-b067-4d85-8141-741e72c8f041" path="/var/lib/kubelet/pods/49caa894-b067-4d85-8141-741e72c8f041/volumes" Nov 02 18:03:28.445295 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:28.445277 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c572702-1be9-42ec-a4b7-243fe8bfd9c8" path="/var/lib/kubelet/pods/9c572702-1be9-42ec-a4b7-243fe8bfd9c8/volumes" Nov 02 18:03:39.185710 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:03:39.185668 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:03:56.031106 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:56.031072 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq"] Nov 02 18:03:56.032217 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:56.032193 4471 cpu_manager.go:414] "containerMap: removing container" podUID="49caa894-b067-4d85-8141-741e72c8f041" containerName="istio-proxy" Nov 02 18:03:56.032217 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:56.032218 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49caa894-b067-4d85-8141-741e72c8f041" containerName="istio-proxy" Nov 02 18:03:56.032487 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:56.032284 4471 cpu_manager.go:414] "containerMap: removing container" podUID="9c572702-1be9-42ec-a4b7-243fe8bfd9c8" containerName="istio-proxy" Nov 02 18:03:56.032487 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:56.032295 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c572702-1be9-42ec-a4b7-243fe8bfd9c8" containerName="istio-proxy" Nov 02 18:03:56.032626 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:56.032609 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c572702-1be9-42ec-a4b7-243fe8bfd9c8" containerName="istio-proxy" Nov 02 18:03:56.032687 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:56.032640 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="49caa894-b067-4d85-8141-741e72c8f041" containerName="istio-proxy" Nov 02 18:03:56.032687 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:56.032657 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="49caa894-b067-4d85-8141-741e72c8f041" containerName="istio-proxy" Nov 02 18:03:56.038635 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:56.038611 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq" Nov 02 18:03:56.064669 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:56.064640 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-557feb-serving-cert\"" Nov 02 18:03:56.072374 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:56.072346 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq"] Nov 02 18:03:56.243327 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:56.239563 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2npjm\" (UniqueName: \"kubernetes.io/projected/54ac053b-bdfa-4471-8799-aabab39ebe94-kube-api-access-2npjm\") pod \"maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq\" (UID: \"54ac053b-bdfa-4471-8799-aabab39ebe94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq" Nov 02 18:03:56.243327 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:56.239875 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54ac053b-bdfa-4471-8799-aabab39ebe94-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq\" (UID: \"54ac053b-bdfa-4471-8799-aabab39ebe94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq" Nov 02 18:03:56.243327 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:56.239987 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54ac053b-bdfa-4471-8799-aabab39ebe94-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq\" (UID: \"54ac053b-bdfa-4471-8799-aabab39ebe94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq" Nov 02 18:03:56.243327 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:56.240093 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54ac053b-bdfa-4471-8799-aabab39ebe94-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq\" (UID: \"54ac053b-bdfa-4471-8799-aabab39ebe94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq" Nov 02 18:03:56.343264 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:56.343169 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2npjm\" (UniqueName: \"kubernetes.io/projected/54ac053b-bdfa-4471-8799-aabab39ebe94-kube-api-access-2npjm\") pod \"maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq\" (UID: \"54ac053b-bdfa-4471-8799-aabab39ebe94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq" Nov 02 18:03:56.343483 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:56.343443 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54ac053b-bdfa-4471-8799-aabab39ebe94-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq\" (UID: \"54ac053b-bdfa-4471-8799-aabab39ebe94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq" Nov 02 18:03:56.343627 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:56.343502 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54ac053b-bdfa-4471-8799-aabab39ebe94-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq\" (UID: \"54ac053b-bdfa-4471-8799-aabab39ebe94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq" Nov 02 18:03:56.343627 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:56.343605 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54ac053b-bdfa-4471-8799-aabab39ebe94-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq\" (UID: \"54ac053b-bdfa-4471-8799-aabab39ebe94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq" Nov 02 18:03:56.343906 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:03:56.343861 4471 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-557feb-serving-cert: secret "maas-api-e2e-shared-a-w3-557feb-serving-cert" not found Nov 02 18:03:56.343988 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:03:56.343951 4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54ac053b-bdfa-4471-8799-aabab39ebe94-maas-api-tls podName:54ac053b-bdfa-4471-8799-aabab39ebe94 nodeName:}" failed. No retries permitted until 2026-11-02 18:03:56.843931658 +0000 UTC m=+2054.539287124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/54ac053b-bdfa-4471-8799-aabab39ebe94-maas-api-tls") pod "maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq" (UID: "54ac053b-bdfa-4471-8799-aabab39ebe94") : secret "maas-api-e2e-shared-a-w3-557feb-serving-cert" not found Nov 02 18:03:56.344922 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:56.344506 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54ac053b-bdfa-4471-8799-aabab39ebe94-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq\" (UID: \"54ac053b-bdfa-4471-8799-aabab39ebe94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq" Nov 02 18:03:56.351767 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:56.351741 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54ac053b-bdfa-4471-8799-aabab39ebe94-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq\" (UID: \"54ac053b-bdfa-4471-8799-aabab39ebe94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq" Nov 02 18:03:56.369027 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:56.369003 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2npjm\" (UniqueName: \"kubernetes.io/projected/54ac053b-bdfa-4471-8799-aabab39ebe94-kube-api-access-2npjm\") pod \"maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq\" (UID: \"54ac053b-bdfa-4471-8799-aabab39ebe94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq" Nov 02 18:03:56.851338 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:56.851293 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54ac053b-bdfa-4471-8799-aabab39ebe94-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq\" (UID: \"54ac053b-bdfa-4471-8799-aabab39ebe94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq" Nov 02 18:03:56.853766 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:56.853735 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54ac053b-bdfa-4471-8799-aabab39ebe94-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq\" (UID: \"54ac053b-bdfa-4471-8799-aabab39ebe94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq" Nov 02 18:03:56.971326 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:56.965592 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq" Nov 02 18:03:57.407509 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:57.378633 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq"] Nov 02 18:03:57.411017 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:03:57.410968 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ac053b_bdfa_4471_8799_aabab39ebe94.slice/crio-b06ee8c131663c2923bde6633a5e8ccafd18dd084dce9bbc2b2e0168be350945 WatchSource:0}: Error finding container b06ee8c131663c2923bde6633a5e8ccafd18dd084dce9bbc2b2e0168be350945: Status 404 returned error can't find the container with id b06ee8c131663c2923bde6633a5e8ccafd18dd084dce9bbc2b2e0168be350945 Nov 02 18:03:57.423321 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:57.419430 4471 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 18:03:57.659750 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:57.659663 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq" event={"ID":"54ac053b-bdfa-4471-8799-aabab39ebe94","Type":"ContainerStarted","Data":"b06ee8c131663c2923bde6633a5e8ccafd18dd084dce9bbc2b2e0168be350945"} Nov 02 18:03:58.670197 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:58.670146 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq" event={"ID":"54ac053b-bdfa-4471-8799-aabab39ebe94","Type":"ContainerStarted","Data":"4a86a253920ec27bf194236708fe19a0fe8421ab2dc30c34b1727e16c8ca6f39"} Nov 02 18:03:58.670715 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:58.670255 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq" Nov 02 18:03:58.718453 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:03:58.718405 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq" podStartSLOduration=3.185900623 podStartE2EDuration="3.718393117s" totalImagesPullingTime="532.492494ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:03:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:03:58.717517777 +0000 UTC m=+2056.412873270" watchObservedRunningTime="2026-11-02 18:03:58.718393117 +0000 UTC m=+2056.413748590" Nov 02 18:04:04.684615 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:04.684585 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq" Nov 02 18:04:09.198361 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:04:09.196470 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:04:12.339610 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:12.339372 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-57d5879ccd-4wcjv"] Nov 02 18:04:12.341641 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:12.341609 4471 cpu_manager.go:414] "containerMap: removing container" podUID="49caa894-b067-4d85-8141-741e72c8f041" containerName="istio-proxy" Nov 02 18:04:12.341828 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:12.341803 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49caa894-b067-4d85-8141-741e72c8f041" containerName="istio-proxy" Nov 02 18:04:12.350363 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:12.350339 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d5879ccd-4wcjv" Nov 02 18:04:12.357810 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:12.357782 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57d5879ccd-4wcjv"] Nov 02 18:04:12.438882 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:12.438829 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b536a1cd-25ed-431a-af71-3cb8a0628c67-console-oauth-config\") pod \"console-57d5879ccd-4wcjv\" (UID: \"b536a1cd-25ed-431a-af71-3cb8a0628c67\") " pod="openshift-console/console-57d5879ccd-4wcjv" Nov 02 18:04:12.439068 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:12.438922 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4cnd\" (UniqueName: \"kubernetes.io/projected/b536a1cd-25ed-431a-af71-3cb8a0628c67-kube-api-access-n4cnd\") pod \"console-57d5879ccd-4wcjv\" (UID: \"b536a1cd-25ed-431a-af71-3cb8a0628c67\") " pod="openshift-console/console-57d5879ccd-4wcjv" Nov 02 18:04:12.439068 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:12.438988 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b536a1cd-25ed-431a-af71-3cb8a0628c67-oauth-serving-cert\") pod \"console-57d5879ccd-4wcjv\" (UID: \"b536a1cd-25ed-431a-af71-3cb8a0628c67\") " pod="openshift-console/console-57d5879ccd-4wcjv" Nov 02 18:04:12.439173 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:12.439097 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b536a1cd-25ed-431a-af71-3cb8a0628c67-console-serving-cert\") pod \"console-57d5879ccd-4wcjv\" (UID: \"b536a1cd-25ed-431a-af71-3cb8a0628c67\") " pod="openshift-console/console-57d5879ccd-4wcjv" Nov 02 18:04:12.439173 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:12.439142 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b536a1cd-25ed-431a-af71-3cb8a0628c67-service-ca\") pod \"console-57d5879ccd-4wcjv\" (UID: \"b536a1cd-25ed-431a-af71-3cb8a0628c67\") " pod="openshift-console/console-57d5879ccd-4wcjv" Nov 02 18:04:12.439291 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:12.439172 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b536a1cd-25ed-431a-af71-3cb8a0628c67-console-config\") pod \"console-57d5879ccd-4wcjv\" (UID: \"b536a1cd-25ed-431a-af71-3cb8a0628c67\") " pod="openshift-console/console-57d5879ccd-4wcjv" Nov 02 18:04:12.439408 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:12.439382 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b536a1cd-25ed-431a-af71-3cb8a0628c67-tmp\") pod \"console-57d5879ccd-4wcjv\" (UID: \"b536a1cd-25ed-431a-af71-3cb8a0628c67\") " pod="openshift-console/console-57d5879ccd-4wcjv" Nov 02 18:04:12.439612 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:12.439586 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b536a1cd-25ed-431a-af71-3cb8a0628c67-trusted-ca-bundle\") pod \"console-57d5879ccd-4wcjv\" (UID: \"b536a1cd-25ed-431a-af71-3cb8a0628c67\") " pod="openshift-console/console-57d5879ccd-4wcjv" Nov 02 18:04:12.541906 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:12.541869 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b536a1cd-25ed-431a-af71-3cb8a0628c67-console-serving-cert\") pod \"console-57d5879ccd-4wcjv\" (UID: \"b536a1cd-25ed-431a-af71-3cb8a0628c67\") " pod="openshift-console/console-57d5879ccd-4wcjv" Nov 02 18:04:12.542100 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:12.541936 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b536a1cd-25ed-431a-af71-3cb8a0628c67-service-ca\") pod \"console-57d5879ccd-4wcjv\" (UID: \"b536a1cd-25ed-431a-af71-3cb8a0628c67\") " pod="openshift-console/console-57d5879ccd-4wcjv" Nov 02 18:04:12.542100 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:12.541969 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b536a1cd-25ed-431a-af71-3cb8a0628c67-console-config\") pod \"console-57d5879ccd-4wcjv\" (UID: \"b536a1cd-25ed-431a-af71-3cb8a0628c67\") " pod="openshift-console/console-57d5879ccd-4wcjv" Nov 02 18:04:12.542100 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:12.542018 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b536a1cd-25ed-431a-af71-3cb8a0628c67-tmp\") pod \"console-57d5879ccd-4wcjv\" (UID: \"b536a1cd-25ed-431a-af71-3cb8a0628c67\") " pod="openshift-console/console-57d5879ccd-4wcjv" Nov 02 18:04:12.542267 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:12.542168 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b536a1cd-25ed-431a-af71-3cb8a0628c67-trusted-ca-bundle\") pod \"console-57d5879ccd-4wcjv\" (UID: \"b536a1cd-25ed-431a-af71-3cb8a0628c67\") " pod="openshift-console/console-57d5879ccd-4wcjv" Nov 02 18:04:12.542568 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:12.542540 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b536a1cd-25ed-431a-af71-3cb8a0628c67-console-oauth-config\") pod \"console-57d5879ccd-4wcjv\" (UID: \"b536a1cd-25ed-431a-af71-3cb8a0628c67\") " pod="openshift-console/console-57d5879ccd-4wcjv" Nov 02 18:04:12.542693 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:12.542619 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4cnd\" (UniqueName: \"kubernetes.io/projected/b536a1cd-25ed-431a-af71-3cb8a0628c67-kube-api-access-n4cnd\") pod \"console-57d5879ccd-4wcjv\" (UID: \"b536a1cd-25ed-431a-af71-3cb8a0628c67\") " pod="openshift-console/console-57d5879ccd-4wcjv" Nov 02 18:04:12.542756 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:12.542690 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b536a1cd-25ed-431a-af71-3cb8a0628c67-oauth-serving-cert\") pod \"console-57d5879ccd-4wcjv\" (UID: \"b536a1cd-25ed-431a-af71-3cb8a0628c67\") " pod="openshift-console/console-57d5879ccd-4wcjv" Nov 02 18:04:12.543063 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:12.543036 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b536a1cd-25ed-431a-af71-3cb8a0628c67-tmp\") pod \"console-57d5879ccd-4wcjv\" (UID: \"b536a1cd-25ed-431a-af71-3cb8a0628c67\") " pod="openshift-console/console-57d5879ccd-4wcjv" Nov 02 18:04:12.546233 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:12.546177 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b536a1cd-25ed-431a-af71-3cb8a0628c67-console-oauth-config\") pod \"console-57d5879ccd-4wcjv\" (UID: \"b536a1cd-25ed-431a-af71-3cb8a0628c67\") " pod="openshift-console/console-57d5879ccd-4wcjv" Nov 02 18:04:12.546891 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:12.546866 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b536a1cd-25ed-431a-af71-3cb8a0628c67-service-ca\") pod \"console-57d5879ccd-4wcjv\" (UID: \"b536a1cd-25ed-431a-af71-3cb8a0628c67\") " pod="openshift-console/console-57d5879ccd-4wcjv" Nov 02 18:04:12.547521 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:12.547499 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b536a1cd-25ed-431a-af71-3cb8a0628c67-console-config\") pod \"console-57d5879ccd-4wcjv\" (UID: \"b536a1cd-25ed-431a-af71-3cb8a0628c67\") " pod="openshift-console/console-57d5879ccd-4wcjv" Nov 02 18:04:12.548163 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:12.548141 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b536a1cd-25ed-431a-af71-3cb8a0628c67-oauth-serving-cert\") pod \"console-57d5879ccd-4wcjv\" (UID: \"b536a1cd-25ed-431a-af71-3cb8a0628c67\") " pod="openshift-console/console-57d5879ccd-4wcjv" Nov 02 18:04:12.548928 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:12.548907 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b536a1cd-25ed-431a-af71-3cb8a0628c67-trusted-ca-bundle\") pod \"console-57d5879ccd-4wcjv\" (UID: \"b536a1cd-25ed-431a-af71-3cb8a0628c67\") " pod="openshift-console/console-57d5879ccd-4wcjv" Nov 02 18:04:12.549628 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:12.549605 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b536a1cd-25ed-431a-af71-3cb8a0628c67-console-serving-cert\") pod \"console-57d5879ccd-4wcjv\" (UID: \"b536a1cd-25ed-431a-af71-3cb8a0628c67\") " pod="openshift-console/console-57d5879ccd-4wcjv" Nov 02 18:04:12.575821 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:12.575787 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4cnd\" (UniqueName: \"kubernetes.io/projected/b536a1cd-25ed-431a-af71-3cb8a0628c67-kube-api-access-n4cnd\") pod \"console-57d5879ccd-4wcjv\" (UID: \"b536a1cd-25ed-431a-af71-3cb8a0628c67\") " pod="openshift-console/console-57d5879ccd-4wcjv" Nov 02 18:04:12.676574 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:12.676480 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d5879ccd-4wcjv" Nov 02 18:04:13.052347 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:13.049235 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57d5879ccd-4wcjv"] Nov 02 18:04:13.877369 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:13.873903 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d5879ccd-4wcjv" event={"ID":"b536a1cd-25ed-431a-af71-3cb8a0628c67","Type":"ContainerStarted","Data":"dddbb32030f6c129993496fbaddb4977ba29089b15f30eb9f898178cf416565a"} Nov 02 18:04:13.877369 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:13.873982 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d5879ccd-4wcjv" event={"ID":"b536a1cd-25ed-431a-af71-3cb8a0628c67","Type":"ContainerStarted","Data":"351a5d6c85c885704aa0fac71e7213ce58a31fddb200fd21c5186d5d6944acf0"} Nov 02 18:04:13.932157 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:13.932092 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-57d5879ccd-4wcjv" podStartSLOduration=1.932072395 podStartE2EDuration="1.932072395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:04:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:04:13.924363916 +0000 UTC m=+2071.619719385" watchObservedRunningTime="2026-11-02 18:04:13.932072395 +0000 UTC m=+2071.627427876" Nov 02 18:04:22.677863 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:22.677830 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57d5879ccd-4wcjv" Nov 02 18:04:22.678262 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:22.677890 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57d5879ccd-4wcjv" Nov 02 18:04:22.682546 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:22.682522 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57d5879ccd-4wcjv" Nov 02 18:04:22.955678 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:22.955607 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57d5879ccd-4wcjv" Nov 02 18:04:23.057697 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:23.055566 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-684568bcb7-h8wc6"] Nov 02 18:04:25.376851 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:25.376817 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rt5cp_1b5ee10f-87a1-4168-904e-ba233f304fe3/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:37.455341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:37.453978 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rt5cp_1b5ee10f-87a1-4168-904e-ba233f304fe3/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:39.186222 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:04:39.186187 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:04:48.083253 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:48.083210 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-684568bcb7-h8wc6" podUID="1e3c97d1-c520-401d-8f11-171f00bba30d" containerName="console" containerID="cri-o://2a4b4e72d24aececf4c66756a2e5bae9079950afe214d5a89e7ed8198d6af52d" gracePeriod=15 Nov 02 18:04:48.329745 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:48.329705 4471 generic.go:356] "Generic (PLEG): container finished" podID="1e3c97d1-c520-401d-8f11-171f00bba30d" containerID="2a4b4e72d24aececf4c66756a2e5bae9079950afe214d5a89e7ed8198d6af52d" exitCode=0 Nov 02 18:04:48.329946 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:48.329766 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-684568bcb7-h8wc6" event={"ID":"1e3c97d1-c520-401d-8f11-171f00bba30d","Type":"ContainerDied","Data":"2a4b4e72d24aececf4c66756a2e5bae9079950afe214d5a89e7ed8198d6af52d"} Nov 02 18:04:48.382267 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:48.382240 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-684568bcb7-h8wc6" Nov 02 18:04:48.453595 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:48.453560 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e3c97d1-c520-401d-8f11-171f00bba30d-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e3c97d1-c520-401d-8f11-171f00bba30d-console-serving-cert\") pod \"1e3c97d1-c520-401d-8f11-171f00bba30d\" (UID: \"1e3c97d1-c520-401d-8f11-171f00bba30d\") " Nov 02 18:04:48.453787 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:48.453609 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e3c97d1-c520-401d-8f11-171f00bba30d-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e3c97d1-c520-401d-8f11-171f00bba30d-service-ca\") pod \"1e3c97d1-c520-401d-8f11-171f00bba30d\" (UID: \"1e3c97d1-c520-401d-8f11-171f00bba30d\") " Nov 02 18:04:48.453787 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:48.453665 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e3c97d1-c520-401d-8f11-171f00bba30d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e3c97d1-c520-401d-8f11-171f00bba30d-trusted-ca-bundle\") pod \"1e3c97d1-c520-401d-8f11-171f00bba30d\" (UID: \"1e3c97d1-c520-401d-8f11-171f00bba30d\") " Nov 02 18:04:48.453787 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:48.453735 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e3c97d1-c520-401d-8f11-171f00bba30d-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e3c97d1-c520-401d-8f11-171f00bba30d-oauth-serving-cert\") pod \"1e3c97d1-c520-401d-8f11-171f00bba30d\" (UID: \"1e3c97d1-c520-401d-8f11-171f00bba30d\") " Nov 02 18:04:48.453787 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:48.453768 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e3c97d1-c520-401d-8f11-171f00bba30d-kube-api-access-xs6pj\" (UniqueName: \"kubernetes.io/projected/1e3c97d1-c520-401d-8f11-171f00bba30d-kube-api-access-xs6pj\") pod \"1e3c97d1-c520-401d-8f11-171f00bba30d\" (UID: \"1e3c97d1-c520-401d-8f11-171f00bba30d\") " Nov 02 18:04:48.454036 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:48.453836 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e3c97d1-c520-401d-8f11-171f00bba30d-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e3c97d1-c520-401d-8f11-171f00bba30d-console-oauth-config\") pod \"1e3c97d1-c520-401d-8f11-171f00bba30d\" (UID: \"1e3c97d1-c520-401d-8f11-171f00bba30d\") " Nov 02 18:04:48.454036 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:48.453977 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e3c97d1-c520-401d-8f11-171f00bba30d-console-config\" (UniqueName: \"kubernetes.io/configmap/1e3c97d1-c520-401d-8f11-171f00bba30d-console-config\") pod \"1e3c97d1-c520-401d-8f11-171f00bba30d\" (UID: \"1e3c97d1-c520-401d-8f11-171f00bba30d\") " Nov 02 18:04:48.454193 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:48.454173 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e3c97d1-c520-401d-8f11-171f00bba30d-tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e3c97d1-c520-401d-8f11-171f00bba30d-tmp\") pod \"1e3c97d1-c520-401d-8f11-171f00bba30d\" (UID: \"1e3c97d1-c520-401d-8f11-171f00bba30d\") " Nov 02 18:04:48.454656 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:48.454628 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e3c97d1-c520-401d-8f11-171f00bba30d-trusted-ca-bundle" pod "1e3c97d1-c520-401d-8f11-171f00bba30d" (UID: "1e3c97d1-c520-401d-8f11-171f00bba30d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:04:48.455156 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:48.455128 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e3c97d1-c520-401d-8f11-171f00bba30d-oauth-serving-cert" pod "1e3c97d1-c520-401d-8f11-171f00bba30d" (UID: "1e3c97d1-c520-401d-8f11-171f00bba30d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:04:48.455618 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:48.455592 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3c97d1-c520-401d-8f11-171f00bba30d-tmp" pod "1e3c97d1-c520-401d-8f11-171f00bba30d" (UID: "1e3c97d1-c520-401d-8f11-171f00bba30d"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:04:48.455854 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:48.455835 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e3c97d1-c520-401d-8f11-171f00bba30d-console-config" pod "1e3c97d1-c520-401d-8f11-171f00bba30d" (UID: "1e3c97d1-c520-401d-8f11-171f00bba30d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:04:48.456232 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:48.456214 4471 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e3c97d1-c520-401d-8f11-171f00bba30d-tmp\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:04:48.456371 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:48.456239 4471 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e3c97d1-c520-401d-8f11-171f00bba30d-trusted-ca-bundle\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:04:48.456371 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:48.456253 4471 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e3c97d1-c520-401d-8f11-171f00bba30d-oauth-serving-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:04:48.456371 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:48.456278 4471 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e3c97d1-c520-401d-8f11-171f00bba30d-console-config\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:04:48.457607 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:48.457584 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e3c97d1-c520-401d-8f11-171f00bba30d-service-ca" pod "1e3c97d1-c520-401d-8f11-171f00bba30d" (UID: "1e3c97d1-c520-401d-8f11-171f00bba30d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:04:48.458539 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:48.458517 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e3c97d1-c520-401d-8f11-171f00bba30d-kube-api-access-xs6pj" pod "1e3c97d1-c520-401d-8f11-171f00bba30d" (UID: "1e3c97d1-c520-401d-8f11-171f00bba30d"). InnerVolumeSpecName "kube-api-access-xs6pj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:04:48.458791 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:48.458769 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e3c97d1-c520-401d-8f11-171f00bba30d-console-oauth-config" pod "1e3c97d1-c520-401d-8f11-171f00bba30d" (UID: "1e3c97d1-c520-401d-8f11-171f00bba30d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:04:48.459043 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:48.459025 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e3c97d1-c520-401d-8f11-171f00bba30d-console-serving-cert" pod "1e3c97d1-c520-401d-8f11-171f00bba30d" (UID: "1e3c97d1-c520-401d-8f11-171f00bba30d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:04:48.557938 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:48.557904 4471 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e3c97d1-c520-401d-8f11-171f00bba30d-console-serving-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:04:48.557938 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:48.557933 4471 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e3c97d1-c520-401d-8f11-171f00bba30d-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:04:48.557938 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:48.557942 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs6pj\" (UniqueName: \"kubernetes.io/projected/1e3c97d1-c520-401d-8f11-171f00bba30d-kube-api-access-xs6pj\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:04:48.558148 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:48.557952 4471 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e3c97d1-c520-401d-8f11-171f00bba30d-console-oauth-config\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:04:49.339842 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:49.339807 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-684568bcb7-h8wc6" event={"ID":"1e3c97d1-c520-401d-8f11-171f00bba30d","Type":"ContainerDied","Data":"cd64fb1b66201ce2022697ebe23e6264bbb866d061b326712b5c0d6e17e9f280"} Nov 02 18:04:49.340301 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:49.339892 4471 scope.go:122] "RemoveContainer" containerID="2a4b4e72d24aececf4c66756a2e5bae9079950afe214d5a89e7ed8198d6af52d" Nov 02 18:04:49.340301 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:49.339817 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-684568bcb7-h8wc6" Nov 02 18:04:49.396792 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:49.396759 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-684568bcb7-h8wc6"] Nov 02 18:04:49.409752 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:49.409722 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-684568bcb7-h8wc6"] Nov 02 18:04:50.446533 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:50.446502 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e3c97d1-c520-401d-8f11-171f00bba30d" path="/var/lib/kubelet/pods/1e3c97d1-c520-401d-8f11-171f00bba30d/volumes" Nov 02 18:04:57.975424 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:57.975346 4471 ???:1] "http: TLS handshake error from 10.0.141.87:34176: EOF" Nov 02 18:04:58.175029 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.174992 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp"] Nov 02 18:04:58.176660 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.176630 4471 cpu_manager.go:414] "containerMap: removing container" podUID="1e3c97d1-c520-401d-8f11-171f00bba30d" containerName="console" Nov 02 18:04:58.176660 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.176658 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e3c97d1-c520-401d-8f11-171f00bba30d" containerName="console" Nov 02 18:04:58.176937 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.176923 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e3c97d1-c520-401d-8f11-171f00bba30d" containerName="console" Nov 02 18:04:58.192640 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.192609 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:04:58.203497 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.203463 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp"] Nov 02 18:04:58.212237 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.212185 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-227939-openshift-default-dockercfg-g6gpq\"" Nov 02 18:04:58.287644 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.287609 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-workload-socket\") pod \"e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:04:58.287800 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.287652 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-workload-certs\") pod \"e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:04:58.287800 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.287674 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-istio-envoy\") pod \"e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:04:58.287909 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.287801 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hv6r\" (UniqueName: \"kubernetes.io/projected/b54fd98c-e8c7-4578-b027-5275834e8c72-kube-api-access-5hv6r\") pod \"e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:04:58.287909 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.287844 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-credential-socket\") pod \"e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:04:58.288047 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.288024 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b54fd98c-e8c7-4578-b027-5275834e8c72-istio-token\") pod \"e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:04:58.288184 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.288161 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b54fd98c-e8c7-4578-b027-5275834e8c72-istio-podinfo\") pod \"e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:04:58.288282 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.288263 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b54fd98c-e8c7-4578-b027-5275834e8c72-istiod-ca-cert\") pod \"e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:04:58.288342 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.288294 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-istio-data\") pod \"e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:04:58.403721 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.403672 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b54fd98c-e8c7-4578-b027-5275834e8c72-istiod-ca-cert\") pod \"e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:04:58.426778 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.426743 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-istio-data\") pod \"e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:04:58.427149 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.427131 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-workload-socket\") pod \"e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:04:58.427271 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.427257 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-workload-certs\") pod \"e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:04:58.427425 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.427407 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-istio-envoy\") pod \"e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:04:58.427573 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.427556 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hv6r\" (UniqueName: \"kubernetes.io/projected/b54fd98c-e8c7-4578-b027-5275834e8c72-kube-api-access-5hv6r\") pod \"e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:04:58.427715 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.427701 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-credential-socket\") pod \"e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:04:58.427970 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.427952 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b54fd98c-e8c7-4578-b027-5275834e8c72-istio-token\") pod \"e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:04:58.428296 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.428276 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b54fd98c-e8c7-4578-b027-5275834e8c72-istio-podinfo\") pod \"e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:04:58.479286 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.428692 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-workload-certs\") pod \"e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:04:58.479534 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.405825 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b54fd98c-e8c7-4578-b027-5275834e8c72-istiod-ca-cert\") pod \"e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:04:58.479645 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.428920 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-istio-data\") pod \"e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:04:58.479729 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.429129 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-workload-socket\") pod \"e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:04:58.479816 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.430000 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-credential-socket\") pod \"e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:04:58.479899 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.447616 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b54fd98c-e8c7-4578-b027-5275834e8c72-istio-podinfo\") pod \"e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:04:58.502323 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.502265 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hv6r\" (UniqueName: \"kubernetes.io/projected/b54fd98c-e8c7-4578-b027-5275834e8c72-kube-api-access-5hv6r\") pod \"e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:04:58.507905 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.507873 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-istio-envoy\") pod \"e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:04:58.528328 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.527168 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b54fd98c-e8c7-4578-b027-5275834e8c72-istio-token\") pod \"e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:04:58.528927 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:58.528900 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:04:59.201268 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:04:59.201239 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb54fd98c_e8c7_4578_b027_5275834e8c72.slice/crio-26d2a8115d26cb1769844f4ef656622e339c98b99f5d4669f7af578d70929da0 WatchSource:0}: Error finding container 26d2a8115d26cb1769844f4ef656622e339c98b99f5d4669f7af578d70929da0: Status 404 returned error can't find the container with id 26d2a8115d26cb1769844f4ef656622e339c98b99f5d4669f7af578d70929da0 Nov 02 18:04:59.204177 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:59.204151 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp"] Nov 02 18:04:59.204525 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:59.204493 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:04:59.204579 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:59.204562 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:04:59.204615 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:59.204591 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:04:59.603705 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:59.603660 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" event={"ID":"b54fd98c-e8c7-4578-b027-5275834e8c72","Type":"ContainerStarted","Data":"e515c5200f7cf6ac9ec59af8a997c85c4400ba615caac97d7a3d496945f2f677"} Nov 02 18:04:59.603877 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:59.603731 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" event={"ID":"b54fd98c-e8c7-4578-b027-5275834e8c72","Type":"ContainerStarted","Data":"26d2a8115d26cb1769844f4ef656622e339c98b99f5d4669f7af578d70929da0"} Nov 02 18:04:59.665219 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:04:59.665158 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" podStartSLOduration=1.665140115 podStartE2EDuration="1.665140115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:04:59.663205117 +0000 UTC m=+2117.358560609" watchObservedRunningTime="2026-11-02 18:04:59.665140115 +0000 UTC m=+2117.360495588" Nov 02 18:05:00.529909 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:00.529877 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:05:00.536025 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:00.535999 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:05:00.687505 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:00.687470 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:05:00.690771 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:00.690746 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:05:09.187229 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:05:09.187189 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:05:11.705095 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:11.705057 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e5cafc4-6nhjm"] Nov 02 18:05:11.710399 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:11.710375 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5cafc4-6nhjm" Nov 02 18:05:11.729263 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:11.729237 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e5cafc4-6nhjm"] Nov 02 18:05:11.735813 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:11.735786 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 18:05:11.735948 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:11.735900 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:05:11.832532 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:11.832492 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf19245a-0076-43fe-934f-e0f8f3ac19c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e5cafc4-6nhjm\" (UID: \"cf19245a-0076-43fe-934f-e0f8f3ac19c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5cafc4-6nhjm" Nov 02 18:05:11.832703 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:11.832688 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf19245a-0076-43fe-934f-e0f8f3ac19c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e5cafc4-6nhjm\" (UID: \"cf19245a-0076-43fe-934f-e0f8f3ac19c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5cafc4-6nhjm" Nov 02 18:05:11.832757 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:11.832744 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77m94\" (UniqueName: \"kubernetes.io/projected/cf19245a-0076-43fe-934f-e0f8f3ac19c4-kube-api-access-77m94\") pod \"maas-api-revoke-sub-3e5cafc4-6nhjm\" (UID: \"cf19245a-0076-43fe-934f-e0f8f3ac19c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5cafc4-6nhjm" Nov 02 18:05:11.935220 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:11.935173 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf19245a-0076-43fe-934f-e0f8f3ac19c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e5cafc4-6nhjm\" (UID: \"cf19245a-0076-43fe-934f-e0f8f3ac19c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5cafc4-6nhjm" Nov 02 18:05:11.935412 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:11.935396 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf19245a-0076-43fe-934f-e0f8f3ac19c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e5cafc4-6nhjm\" (UID: \"cf19245a-0076-43fe-934f-e0f8f3ac19c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5cafc4-6nhjm" Nov 02 18:05:11.935470 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:11.935459 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77m94\" (UniqueName: \"kubernetes.io/projected/cf19245a-0076-43fe-934f-e0f8f3ac19c4-kube-api-access-77m94\") pod \"maas-api-revoke-sub-3e5cafc4-6nhjm\" (UID: \"cf19245a-0076-43fe-934f-e0f8f3ac19c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5cafc4-6nhjm" Nov 02 18:05:11.936104 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:11.936072 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf19245a-0076-43fe-934f-e0f8f3ac19c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e5cafc4-6nhjm\" (UID: \"cf19245a-0076-43fe-934f-e0f8f3ac19c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5cafc4-6nhjm" Nov 02 18:05:11.938083 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:11.938062 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf19245a-0076-43fe-934f-e0f8f3ac19c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e5cafc4-6nhjm\" (UID: \"cf19245a-0076-43fe-934f-e0f8f3ac19c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5cafc4-6nhjm" Nov 02 18:05:11.988535 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:11.988474 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77m94\" (UniqueName: \"kubernetes.io/projected/cf19245a-0076-43fe-934f-e0f8f3ac19c4-kube-api-access-77m94\") pod \"maas-api-revoke-sub-3e5cafc4-6nhjm\" (UID: \"cf19245a-0076-43fe-934f-e0f8f3ac19c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5cafc4-6nhjm" Nov 02 18:05:12.028283 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:12.028250 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5cafc4-6nhjm" Nov 02 18:05:12.347596 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:05:12.347561 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf19245a_0076_43fe_934f_e0f8f3ac19c4.slice/crio-83190bc22619cfb25ca37cb55e5ae7cca3f684be3436fa268bcab024ad249e10 WatchSource:0}: Error finding container 83190bc22619cfb25ca37cb55e5ae7cca3f684be3436fa268bcab024ad249e10: Status 404 returned error can't find the container with id 83190bc22619cfb25ca37cb55e5ae7cca3f684be3436fa268bcab024ad249e10 Nov 02 18:05:12.350013 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:12.349985 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e5cafc4-6nhjm"] Nov 02 18:05:12.505618 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:12.505583 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4221592b-cktkl"] Nov 02 18:05:12.512190 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:12.512167 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4221592b-cktkl" Nov 02 18:05:12.529379 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:12.529345 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4221592b-cktkl"] Nov 02 18:05:12.544504 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:12.544469 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ldqm\" (UniqueName: \"kubernetes.io/projected/517d2554-a194-476c-a192-d60a6675e38a-kube-api-access-8ldqm\") pod \"maas-api-revoke-sub-4221592b-cktkl\" (UID: \"517d2554-a194-476c-a192-d60a6675e38a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4221592b-cktkl" Nov 02 18:05:12.544640 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:12.544522 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/517d2554-a194-476c-a192-d60a6675e38a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4221592b-cktkl\" (UID: \"517d2554-a194-476c-a192-d60a6675e38a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4221592b-cktkl" Nov 02 18:05:12.544883 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:12.544855 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/517d2554-a194-476c-a192-d60a6675e38a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4221592b-cktkl\" (UID: \"517d2554-a194-476c-a192-d60a6675e38a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4221592b-cktkl" Nov 02 18:05:12.647805 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:12.647712 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/517d2554-a194-476c-a192-d60a6675e38a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4221592b-cktkl\" (UID: \"517d2554-a194-476c-a192-d60a6675e38a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4221592b-cktkl" Nov 02 18:05:12.649178 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:12.649151 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ldqm\" (UniqueName: \"kubernetes.io/projected/517d2554-a194-476c-a192-d60a6675e38a-kube-api-access-8ldqm\") pod \"maas-api-revoke-sub-4221592b-cktkl\" (UID: \"517d2554-a194-476c-a192-d60a6675e38a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4221592b-cktkl" Nov 02 18:05:12.649645 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:12.649622 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/517d2554-a194-476c-a192-d60a6675e38a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4221592b-cktkl\" (UID: \"517d2554-a194-476c-a192-d60a6675e38a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4221592b-cktkl" Nov 02 18:05:12.650766 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:12.650747 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/517d2554-a194-476c-a192-d60a6675e38a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4221592b-cktkl\" (UID: \"517d2554-a194-476c-a192-d60a6675e38a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4221592b-cktkl" Nov 02 18:05:12.653575 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:12.653534 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/517d2554-a194-476c-a192-d60a6675e38a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4221592b-cktkl\" (UID: \"517d2554-a194-476c-a192-d60a6675e38a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4221592b-cktkl" Nov 02 18:05:12.673046 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:12.673025 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ldqm\" (UniqueName: \"kubernetes.io/projected/517d2554-a194-476c-a192-d60a6675e38a-kube-api-access-8ldqm\") pod \"maas-api-revoke-sub-4221592b-cktkl\" (UID: \"517d2554-a194-476c-a192-d60a6675e38a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4221592b-cktkl" Nov 02 18:05:12.828622 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:12.828594 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4221592b-cktkl" Nov 02 18:05:12.876615 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:12.876519 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5cafc4-6nhjm" event={"ID":"cf19245a-0076-43fe-934f-e0f8f3ac19c4","Type":"ContainerStarted","Data":"83190bc22619cfb25ca37cb55e5ae7cca3f684be3436fa268bcab024ad249e10"} Nov 02 18:05:13.142828 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:05:13.142791 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod517d2554_a194_476c_a192_d60a6675e38a.slice/crio-e8db036845e7db39bd75534a3db86f7379522c619d99b838644620d4416f8b0f WatchSource:0}: Error finding container e8db036845e7db39bd75534a3db86f7379522c619d99b838644620d4416f8b0f: Status 404 returned error can't find the container with id e8db036845e7db39bd75534a3db86f7379522c619d99b838644620d4416f8b0f Nov 02 18:05:13.147348 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:13.147320 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4221592b-cktkl"] Nov 02 18:05:13.890109 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:13.890067 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4221592b-cktkl" event={"ID":"517d2554-a194-476c-a192-d60a6675e38a","Type":"ContainerStarted","Data":"be7de622429723d40162e60b9a5cc831063246abf102b08690692d47c0252773"} Nov 02 18:05:13.890632 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:13.890133 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4221592b-cktkl" event={"ID":"517d2554-a194-476c-a192-d60a6675e38a","Type":"ContainerStarted","Data":"e8db036845e7db39bd75534a3db86f7379522c619d99b838644620d4416f8b0f"} Nov 02 18:05:13.892073 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:13.892042 4471 generic.go:356] "Generic (PLEG): container finished" podID="cf19245a-0076-43fe-934f-e0f8f3ac19c4" containerID="a4448c42d1a716a6e6b50f1746bbd530359edc623fb4fd91fed9aeb3df020fa3" exitCode=0 Nov 02 18:05:13.892199 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:13.892127 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5cafc4-6nhjm" event={"ID":"cf19245a-0076-43fe-934f-e0f8f3ac19c4","Type":"ContainerDied","Data":"a4448c42d1a716a6e6b50f1746bbd530359edc623fb4fd91fed9aeb3df020fa3"} Nov 02 18:05:13.938959 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:13.938900 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4221592b-cktkl" podStartSLOduration=1.938880084 podStartE2EDuration="1.938880084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:05:13.937459216 +0000 UTC m=+2131.632814716" watchObservedRunningTime="2026-11-02 18:05:13.938880084 +0000 UTC m=+2131.634235558" Nov 02 18:05:14.908189 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:14.908156 4471 generic.go:356] "Generic (PLEG): container finished" podID="517d2554-a194-476c-a192-d60a6675e38a" containerID="be7de622429723d40162e60b9a5cc831063246abf102b08690692d47c0252773" exitCode=0 Nov 02 18:05:14.908604 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:14.908239 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4221592b-cktkl" event={"ID":"517d2554-a194-476c-a192-d60a6675e38a","Type":"ContainerDied","Data":"be7de622429723d40162e60b9a5cc831063246abf102b08690692d47c0252773"} Nov 02 18:05:15.179990 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:15.179961 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5cafc4-6nhjm" Nov 02 18:05:15.252445 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:15.252409 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf19245a-0076-43fe-934f-e0f8f3ac19c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf19245a-0076-43fe-934f-e0f8f3ac19c4-maas-api-xks-ca\") pod \"cf19245a-0076-43fe-934f-e0f8f3ac19c4\" (UID: \"cf19245a-0076-43fe-934f-e0f8f3ac19c4\") " Nov 02 18:05:15.252624 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:15.252572 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf19245a-0076-43fe-934f-e0f8f3ac19c4-kube-api-access-77m94\" (UniqueName: \"kubernetes.io/projected/cf19245a-0076-43fe-934f-e0f8f3ac19c4-kube-api-access-77m94\") pod \"cf19245a-0076-43fe-934f-e0f8f3ac19c4\" (UID: \"cf19245a-0076-43fe-934f-e0f8f3ac19c4\") " Nov 02 18:05:15.252785 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:15.252768 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf19245a-0076-43fe-934f-e0f8f3ac19c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf19245a-0076-43fe-934f-e0f8f3ac19c4-maas-api-openshift-ca\") pod \"cf19245a-0076-43fe-934f-e0f8f3ac19c4\" (UID: \"cf19245a-0076-43fe-934f-e0f8f3ac19c4\") " Nov 02 18:05:15.253155 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:15.253126 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf19245a-0076-43fe-934f-e0f8f3ac19c4-maas-api-openshift-ca" pod "cf19245a-0076-43fe-934f-e0f8f3ac19c4" (UID: "cf19245a-0076-43fe-934f-e0f8f3ac19c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:05:15.253836 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:15.253805 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf19245a-0076-43fe-934f-e0f8f3ac19c4-maas-api-openshift-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:15.255068 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:15.255038 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf19245a-0076-43fe-934f-e0f8f3ac19c4-kube-api-access-77m94" pod "cf19245a-0076-43fe-934f-e0f8f3ac19c4" (UID: "cf19245a-0076-43fe-934f-e0f8f3ac19c4"). InnerVolumeSpecName "kube-api-access-77m94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:05:15.258985 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:15.258958 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf19245a-0076-43fe-934f-e0f8f3ac19c4-maas-api-xks-ca" pod "cf19245a-0076-43fe-934f-e0f8f3ac19c4" (UID: "cf19245a-0076-43fe-934f-e0f8f3ac19c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:05:15.355962 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:15.355918 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf19245a-0076-43fe-934f-e0f8f3ac19c4-maas-api-xks-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:15.355962 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:15.355955 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77m94\" (UniqueName: \"kubernetes.io/projected/cf19245a-0076-43fe-934f-e0f8f3ac19c4-kube-api-access-77m94\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:15.927374 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:15.927185 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5cafc4-6nhjm" Nov 02 18:05:15.929433 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:15.929398 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5cafc4-6nhjm" event={"ID":"cf19245a-0076-43fe-934f-e0f8f3ac19c4","Type":"ContainerDied","Data":"83190bc22619cfb25ca37cb55e5ae7cca3f684be3436fa268bcab024ad249e10"} Nov 02 18:05:15.929433 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:15.929445 4471 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83190bc22619cfb25ca37cb55e5ae7cca3f684be3436fa268bcab024ad249e10" Nov 02 18:05:16.171731 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:16.171702 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4221592b-cktkl" Nov 02 18:05:16.293549 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:16.293512 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/517d2554-a194-476c-a192-d60a6675e38a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/517d2554-a194-476c-a192-d60a6675e38a-maas-api-openshift-ca\") pod \"517d2554-a194-476c-a192-d60a6675e38a\" (UID: \"517d2554-a194-476c-a192-d60a6675e38a\") " Nov 02 18:05:16.293731 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:16.293560 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/517d2554-a194-476c-a192-d60a6675e38a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/517d2554-a194-476c-a192-d60a6675e38a-maas-api-xks-ca\") pod \"517d2554-a194-476c-a192-d60a6675e38a\" (UID: \"517d2554-a194-476c-a192-d60a6675e38a\") " Nov 02 18:05:16.293731 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:16.293646 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/517d2554-a194-476c-a192-d60a6675e38a-kube-api-access-8ldqm\" (UniqueName: \"kubernetes.io/projected/517d2554-a194-476c-a192-d60a6675e38a-kube-api-access-8ldqm\") pod \"517d2554-a194-476c-a192-d60a6675e38a\" (UID: \"517d2554-a194-476c-a192-d60a6675e38a\") " Nov 02 18:05:16.293982 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:16.293951 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/517d2554-a194-476c-a192-d60a6675e38a-maas-api-openshift-ca" pod "517d2554-a194-476c-a192-d60a6675e38a" (UID: "517d2554-a194-476c-a192-d60a6675e38a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:05:16.294661 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:16.294633 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/517d2554-a194-476c-a192-d60a6675e38a-maas-api-openshift-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:16.295943 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:16.295912 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/517d2554-a194-476c-a192-d60a6675e38a-maas-api-xks-ca" pod "517d2554-a194-476c-a192-d60a6675e38a" (UID: "517d2554-a194-476c-a192-d60a6675e38a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:05:16.300512 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:16.300484 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/517d2554-a194-476c-a192-d60a6675e38a-kube-api-access-8ldqm" pod "517d2554-a194-476c-a192-d60a6675e38a" (UID: "517d2554-a194-476c-a192-d60a6675e38a"). InnerVolumeSpecName "kube-api-access-8ldqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:05:16.396421 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:16.396388 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/517d2554-a194-476c-a192-d60a6675e38a-maas-api-xks-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:16.396421 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:16.396418 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ldqm\" (UniqueName: \"kubernetes.io/projected/517d2554-a194-476c-a192-d60a6675e38a-kube-api-access-8ldqm\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:16.945843 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:16.945710 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4221592b-cktkl" event={"ID":"517d2554-a194-476c-a192-d60a6675e38a","Type":"ContainerDied","Data":"e8db036845e7db39bd75534a3db86f7379522c619d99b838644620d4416f8b0f"} Nov 02 18:05:16.945843 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:16.945699 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4221592b-cktkl" Nov 02 18:05:16.945843 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:16.945800 4471 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8db036845e7db39bd75534a3db86f7379522c619d99b838644620d4416f8b0f" Nov 02 18:05:18.941104 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:18.941021 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-8564db559-mnvjc"] Nov 02 18:05:18.944733 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:18.944543 4471 cpu_manager.go:414] "containerMap: removing container" podUID="cf19245a-0076-43fe-934f-e0f8f3ac19c4" containerName="revoke-keys" Nov 02 18:05:18.945072 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:18.945052 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf19245a-0076-43fe-934f-e0f8f3ac19c4" containerName="revoke-keys" Nov 02 18:05:18.945336 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:18.945297 4471 cpu_manager.go:414] "containerMap: removing container" podUID="517d2554-a194-476c-a192-d60a6675e38a" containerName="revoke-keys" Nov 02 18:05:18.945491 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:18.945458 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="517d2554-a194-476c-a192-d60a6675e38a" containerName="revoke-keys" Nov 02 18:05:18.948292 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:18.946033 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf19245a-0076-43fe-934f-e0f8f3ac19c4" containerName="revoke-keys" Nov 02 18:05:18.948292 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:18.946071 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="517d2554-a194-476c-a192-d60a6675e38a" containerName="revoke-keys" Nov 02 18:05:19.015664 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:18.999125 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8564db559-mnvjc" Nov 02 18:05:19.015664 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:19.000897 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8564db559-mnvjc"] Nov 02 18:05:19.119888 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:19.119846 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-service-ca\") pod \"console-8564db559-mnvjc\" (UID: \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\") " pod="openshift-console/console-8564db559-mnvjc" Nov 02 18:05:19.120068 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:19.119909 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-tmp\") pod \"console-8564db559-mnvjc\" (UID: \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\") " pod="openshift-console/console-8564db559-mnvjc" Nov 02 18:05:19.120068 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:19.120035 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-oauth-serving-cert\") pod \"console-8564db559-mnvjc\" (UID: \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\") " pod="openshift-console/console-8564db559-mnvjc" Nov 02 18:05:19.120177 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:19.120068 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-console-config\") pod \"console-8564db559-mnvjc\" (UID: \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\") " pod="openshift-console/console-8564db559-mnvjc" Nov 02 18:05:19.120177 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:19.120101 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-trusted-ca-bundle\") pod \"console-8564db559-mnvjc\" (UID: \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\") " pod="openshift-console/console-8564db559-mnvjc" Nov 02 18:05:19.120293 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:19.120198 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89fl2\" (UniqueName: \"kubernetes.io/projected/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-kube-api-access-89fl2\") pod \"console-8564db559-mnvjc\" (UID: \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\") " pod="openshift-console/console-8564db559-mnvjc" Nov 02 18:05:19.120363 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:19.120293 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-console-oauth-config\") pod \"console-8564db559-mnvjc\" (UID: \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\") " pod="openshift-console/console-8564db559-mnvjc" Nov 02 18:05:19.120445 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:19.120429 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-console-serving-cert\") pod \"console-8564db559-mnvjc\" (UID: \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\") " pod="openshift-console/console-8564db559-mnvjc" Nov 02 18:05:19.222879 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:19.222787 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-console-serving-cert\") pod \"console-8564db559-mnvjc\" (UID: \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\") " pod="openshift-console/console-8564db559-mnvjc" Nov 02 18:05:19.223156 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:19.223118 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-service-ca\") pod \"console-8564db559-mnvjc\" (UID: \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\") " pod="openshift-console/console-8564db559-mnvjc" Nov 02 18:05:19.223268 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:19.223183 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-tmp\") pod \"console-8564db559-mnvjc\" (UID: \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\") " pod="openshift-console/console-8564db559-mnvjc" Nov 02 18:05:19.223268 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:19.223252 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-oauth-serving-cert\") pod \"console-8564db559-mnvjc\" (UID: \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\") " pod="openshift-console/console-8564db559-mnvjc" Nov 02 18:05:19.223406 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:19.223281 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-console-config\") pod \"console-8564db559-mnvjc\" (UID: \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\") " pod="openshift-console/console-8564db559-mnvjc" Nov 02 18:05:19.223406 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:19.223324 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-trusted-ca-bundle\") pod \"console-8564db559-mnvjc\" (UID: \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\") " pod="openshift-console/console-8564db559-mnvjc" Nov 02 18:05:19.223406 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:19.223352 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89fl2\" (UniqueName: \"kubernetes.io/projected/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-kube-api-access-89fl2\") pod \"console-8564db559-mnvjc\" (UID: \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\") " pod="openshift-console/console-8564db559-mnvjc" Nov 02 18:05:19.223406 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:19.223401 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-console-oauth-config\") pod \"console-8564db559-mnvjc\" (UID: \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\") " pod="openshift-console/console-8564db559-mnvjc" Nov 02 18:05:19.224665 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:19.224631 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-trusted-ca-bundle\") pod \"console-8564db559-mnvjc\" (UID: \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\") " pod="openshift-console/console-8564db559-mnvjc" Nov 02 18:05:19.225147 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:19.225125 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-console-config\") pod \"console-8564db559-mnvjc\" (UID: \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\") " pod="openshift-console/console-8564db559-mnvjc" Nov 02 18:05:19.225927 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:19.225908 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-service-ca\") pod \"console-8564db559-mnvjc\" (UID: \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\") " pod="openshift-console/console-8564db559-mnvjc" Nov 02 18:05:19.226827 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:19.226808 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-tmp\") pod \"console-8564db559-mnvjc\" (UID: \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\") " pod="openshift-console/console-8564db559-mnvjc" Nov 02 18:05:19.227660 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:19.227639 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-oauth-serving-cert\") pod \"console-8564db559-mnvjc\" (UID: \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\") " pod="openshift-console/console-8564db559-mnvjc" Nov 02 18:05:19.234060 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:19.234028 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-console-oauth-config\") pod \"console-8564db559-mnvjc\" (UID: \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\") " pod="openshift-console/console-8564db559-mnvjc" Nov 02 18:05:19.235416 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:19.235253 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-console-serving-cert\") pod \"console-8564db559-mnvjc\" (UID: \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\") " pod="openshift-console/console-8564db559-mnvjc" Nov 02 18:05:19.259925 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:19.259895 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89fl2\" (UniqueName: \"kubernetes.io/projected/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-kube-api-access-89fl2\") pod \"console-8564db559-mnvjc\" (UID: \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\") " pod="openshift-console/console-8564db559-mnvjc" Nov 02 18:05:19.345911 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:19.345878 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8564db559-mnvjc" Nov 02 18:05:19.743680 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:19.739507 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8564db559-mnvjc"] Nov 02 18:05:20.064940 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:20.064903 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8564db559-mnvjc" event={"ID":"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe","Type":"ContainerStarted","Data":"44c733729df52fe4036eca347a2eedbfcd380e5a6a2150f22f3b92084cfd237a"} Nov 02 18:05:20.065392 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:20.064971 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8564db559-mnvjc" event={"ID":"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe","Type":"ContainerStarted","Data":"3fed70d86bf78af91d6cc98f7b4bbba82168129585dafb2c56d16d6c02a59299"} Nov 02 18:05:20.114392 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:20.114338 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-8564db559-mnvjc" podStartSLOduration=2.114319282 podStartE2EDuration="2.114319282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:05:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:05:20.109818813 +0000 UTC m=+2137.805174308" watchObservedRunningTime="2026-11-02 18:05:20.114319282 +0000 UTC m=+2137.809674768" Nov 02 18:05:21.298291 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:21.298209 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2"] Nov 02 18:05:21.299272 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:21.299210 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" podUID="81708d59-e82f-4259-9bf1-cec029c05c73" containerName="istio-proxy" containerID="cri-o://9d0c192800201632bd49dc7cdd0360bdce4d1074c206f02d30b64d3baf2c16a9" gracePeriod=30 Nov 02 18:05:24.888740 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:24.888698 4471 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:05:24.889149 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:24.888772 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" podUID="81708d59-e82f-4259-9bf1-cec029c05c73" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:05:26.551787 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:26.551760 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 18:05:26.647639 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:26.647541 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81708d59-e82f-4259-9bf1-cec029c05c73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81708d59-e82f-4259-9bf1-cec029c05c73-istio-podinfo\") pod \"81708d59-e82f-4259-9bf1-cec029c05c73\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " Nov 02 18:05:26.647820 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:26.647652 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-workload-certs\") pod \"81708d59-e82f-4259-9bf1-cec029c05c73\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " Nov 02 18:05:26.647820 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:26.647809 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81708d59-e82f-4259-9bf1-cec029c05c73-kube-api-access-6sdj5\" (UniqueName: \"kubernetes.io/projected/81708d59-e82f-4259-9bf1-cec029c05c73-kube-api-access-6sdj5\") pod \"81708d59-e82f-4259-9bf1-cec029c05c73\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " Nov 02 18:05:26.647947 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:26.647852 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-istio-envoy\") pod \"81708d59-e82f-4259-9bf1-cec029c05c73\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " Nov 02 18:05:26.647947 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:26.647919 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-workload-socket\") pod \"81708d59-e82f-4259-9bf1-cec029c05c73\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " Nov 02 18:05:26.648068 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:26.647963 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81708d59-e82f-4259-9bf1-cec029c05c73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81708d59-e82f-4259-9bf1-cec029c05c73-istiod-ca-cert\") pod \"81708d59-e82f-4259-9bf1-cec029c05c73\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " Nov 02 18:05:26.648068 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:26.647967 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-workload-certs" pod "81708d59-e82f-4259-9bf1-cec029c05c73" (UID: "81708d59-e82f-4259-9bf1-cec029c05c73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:26.648179 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:26.648086 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81708d59-e82f-4259-9bf1-cec029c05c73-istio-token\" (UniqueName: \"kubernetes.io/projected/81708d59-e82f-4259-9bf1-cec029c05c73-istio-token\") pod \"81708d59-e82f-4259-9bf1-cec029c05c73\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " Nov 02 18:05:26.648179 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:26.648171 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-istio-data\") pod \"81708d59-e82f-4259-9bf1-cec029c05c73\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " Nov 02 18:05:26.648319 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:26.648208 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-workload-socket" pod "81708d59-e82f-4259-9bf1-cec029c05c73" (UID: "81708d59-e82f-4259-9bf1-cec029c05c73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:26.648319 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:26.648214 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-credential-socket\") pod \"81708d59-e82f-4259-9bf1-cec029c05c73\" (UID: \"81708d59-e82f-4259-9bf1-cec029c05c73\") " Nov 02 18:05:26.649495 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:26.648534 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-istio-data" pod "81708d59-e82f-4259-9bf1-cec029c05c73" (UID: "81708d59-e82f-4259-9bf1-cec029c05c73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:26.649495 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:26.648749 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-credential-socket" pod "81708d59-e82f-4259-9bf1-cec029c05c73" (UID: "81708d59-e82f-4259-9bf1-cec029c05c73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:26.649495 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:26.648772 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81708d59-e82f-4259-9bf1-cec029c05c73-istiod-ca-cert" pod "81708d59-e82f-4259-9bf1-cec029c05c73" (UID: "81708d59-e82f-4259-9bf1-cec029c05c73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:05:26.649495 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:26.649396 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:26.649495 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:26.649421 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81708d59-e82f-4259-9bf1-cec029c05c73-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:26.649495 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:26.649437 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:26.649495 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:26.649452 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:26.649495 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:26.649467 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:26.650497 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:26.650465 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81708d59-e82f-4259-9bf1-cec029c05c73-istio-token" pod "81708d59-e82f-4259-9bf1-cec029c05c73" (UID: "81708d59-e82f-4259-9bf1-cec029c05c73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:05:26.650604 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:26.650522 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81708d59-e82f-4259-9bf1-cec029c05c73-istio-podinfo" pod "81708d59-e82f-4259-9bf1-cec029c05c73" (UID: "81708d59-e82f-4259-9bf1-cec029c05c73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:05:26.652287 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:26.652254 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-istio-envoy" pod "81708d59-e82f-4259-9bf1-cec029c05c73" (UID: "81708d59-e82f-4259-9bf1-cec029c05c73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:26.652778 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:26.652738 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81708d59-e82f-4259-9bf1-cec029c05c73-kube-api-access-6sdj5" pod "81708d59-e82f-4259-9bf1-cec029c05c73" (UID: "81708d59-e82f-4259-9bf1-cec029c05c73"). InnerVolumeSpecName "kube-api-access-6sdj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:05:26.751593 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:26.751557 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81708d59-e82f-4259-9bf1-cec029c05c73-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:26.751593 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:26.751588 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sdj5\" (UniqueName: \"kubernetes.io/projected/81708d59-e82f-4259-9bf1-cec029c05c73-kube-api-access-6sdj5\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:26.751593 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:26.751599 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81708d59-e82f-4259-9bf1-cec029c05c73-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:26.751838 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:26.751608 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81708d59-e82f-4259-9bf1-cec029c05c73-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:27.129495 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:27.129458 4471 generic.go:356] "Generic (PLEG): container finished" podID="81708d59-e82f-4259-9bf1-cec029c05c73" containerID="9d0c192800201632bd49dc7cdd0360bdce4d1074c206f02d30b64d3baf2c16a9" exitCode=0 Nov 02 18:05:27.129672 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:27.129500 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" Nov 02 18:05:27.129672 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:27.129548 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" event={"ID":"81708d59-e82f-4259-9bf1-cec029c05c73","Type":"ContainerDied","Data":"9d0c192800201632bd49dc7cdd0360bdce4d1074c206f02d30b64d3baf2c16a9"} Nov 02 18:05:27.129672 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:27.129602 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2" event={"ID":"81708d59-e82f-4259-9bf1-cec029c05c73","Type":"ContainerDied","Data":"6777cc5d2ffa9d099ad40ff20a5234949cdd1153c04eca35f36dc548c4c4062c"} Nov 02 18:05:27.129672 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:27.129630 4471 scope.go:122] "RemoveContainer" containerID="9d0c192800201632bd49dc7cdd0360bdce4d1074c206f02d30b64d3baf2c16a9" Nov 02 18:05:27.142662 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:27.142641 4471 scope.go:122] "RemoveContainer" containerID="9d0c192800201632bd49dc7cdd0360bdce4d1074c206f02d30b64d3baf2c16a9" Nov 02 18:05:27.142938 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:05:27.142918 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d0c192800201632bd49dc7cdd0360bdce4d1074c206f02d30b64d3baf2c16a9\": container with ID starting with 9d0c192800201632bd49dc7cdd0360bdce4d1074c206f02d30b64d3baf2c16a9 not found: ID does not exist" containerID="9d0c192800201632bd49dc7cdd0360bdce4d1074c206f02d30b64d3baf2c16a9" Nov 02 18:05:27.142978 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:27.142947 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d0c192800201632bd49dc7cdd0360bdce4d1074c206f02d30b64d3baf2c16a9"} err="failed to get container status \"9d0c192800201632bd49dc7cdd0360bdce4d1074c206f02d30b64d3baf2c16a9\": rpc error: code = NotFound desc = could not find container \"9d0c192800201632bd49dc7cdd0360bdce4d1074c206f02d30b64d3baf2c16a9\": container with ID starting with 9d0c192800201632bd49dc7cdd0360bdce4d1074c206f02d30b64d3baf2c16a9 not found: ID does not exist" Nov 02 18:05:27.201063 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:27.201023 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2"] Nov 02 18:05:27.213497 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:27.213464 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-bj9m2"] Nov 02 18:05:28.446621 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:28.446581 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81708d59-e82f-4259-9bf1-cec029c05c73" path="/var/lib/kubelet/pods/81708d59-e82f-4259-9bf1-cec029c05c73/volumes" Nov 02 18:05:29.346272 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:29.346231 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8564db559-mnvjc" Nov 02 18:05:29.346516 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:29.346338 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8564db559-mnvjc" Nov 02 18:05:29.351169 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:29.351146 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8564db559-mnvjc" Nov 02 18:05:30.184550 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:30.184516 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8564db559-mnvjc" Nov 02 18:05:30.275106 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:30.275070 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57d5879ccd-4wcjv"] Nov 02 18:05:30.617150 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:30.617105 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm"] Nov 02 18:05:31.943257 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:31.943216 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5"] Nov 02 18:05:32.531171 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:32.531127 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c"] Nov 02 18:05:33.002377 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:33.002266 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn"] Nov 02 18:05:39.255284 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:39.255246 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx"] Nov 02 18:05:39.331644 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:05:39.331553 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:05:39.782760 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:39.782702 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw"] Nov 02 18:05:45.645380 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:45.645332 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm" podUID="a4d08194-c737-4a26-9198-ecfd8e6554f0" containerName="main" containerID="cri-o://89cd0366ba9b02dc7892197fca2c1426b66ef294d90b91a736cffb13bc3932ff" gracePeriod=45 Nov 02 18:05:45.942014 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:45.941988 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm" Nov 02 18:05:46.046393 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:46.046353 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d08194-c737-4a26-9198-ecfd8e6554f0-home\" (UniqueName: \"kubernetes.io/empty-dir/a4d08194-c737-4a26-9198-ecfd8e6554f0-home\") pod \"a4d08194-c737-4a26-9198-ecfd8e6554f0\" (UID: \"a4d08194-c737-4a26-9198-ecfd8e6554f0\") " Nov 02 18:05:46.046625 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:46.046603 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4d08194-c737-4a26-9198-ecfd8e6554f0-kube-api-access-ctdlh\" (UniqueName: \"kubernetes.io/projected/a4d08194-c737-4a26-9198-ecfd8e6554f0-kube-api-access-ctdlh\") pod \"a4d08194-c737-4a26-9198-ecfd8e6554f0\" (UID: \"a4d08194-c737-4a26-9198-ecfd8e6554f0\") " Nov 02 18:05:46.046689 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:46.046662 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4d08194-c737-4a26-9198-ecfd8e6554f0-tls-certs\" (UniqueName: \"kubernetes.io/secret/a4d08194-c737-4a26-9198-ecfd8e6554f0-tls-certs\") pod \"a4d08194-c737-4a26-9198-ecfd8e6554f0\" (UID: \"a4d08194-c737-4a26-9198-ecfd8e6554f0\") " Nov 02 18:05:46.046744 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:46.046704 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d08194-c737-4a26-9198-ecfd8e6554f0-home" pod "a4d08194-c737-4a26-9198-ecfd8e6554f0" (UID: "a4d08194-c737-4a26-9198-ecfd8e6554f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:46.046789 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:46.046775 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d08194-c737-4a26-9198-ecfd8e6554f0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4d08194-c737-4a26-9198-ecfd8e6554f0-tmp-dir\") pod \"a4d08194-c737-4a26-9198-ecfd8e6554f0\" (UID: \"a4d08194-c737-4a26-9198-ecfd8e6554f0\") " Nov 02 18:05:46.046936 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:46.046919 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d08194-c737-4a26-9198-ecfd8e6554f0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4d08194-c737-4a26-9198-ecfd8e6554f0-model-cache\") pod \"a4d08194-c737-4a26-9198-ecfd8e6554f0\" (UID: \"a4d08194-c737-4a26-9198-ecfd8e6554f0\") " Nov 02 18:05:46.047022 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:46.046995 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d08194-c737-4a26-9198-ecfd8e6554f0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4d08194-c737-4a26-9198-ecfd8e6554f0-dshm\") pod \"a4d08194-c737-4a26-9198-ecfd8e6554f0\" (UID: \"a4d08194-c737-4a26-9198-ecfd8e6554f0\") " Nov 02 18:05:46.047372 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:46.047342 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d08194-c737-4a26-9198-ecfd8e6554f0-model-cache" pod "a4d08194-c737-4a26-9198-ecfd8e6554f0" (UID: "a4d08194-c737-4a26-9198-ecfd8e6554f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:46.047969 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:46.047945 4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4d08194-c737-4a26-9198-ecfd8e6554f0-model-cache\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:46.048068 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:46.047974 4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4d08194-c737-4a26-9198-ecfd8e6554f0-home\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:46.048286 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:46.048253 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d08194-c737-4a26-9198-ecfd8e6554f0-tmp-dir" pod "a4d08194-c737-4a26-9198-ecfd8e6554f0" (UID: "a4d08194-c737-4a26-9198-ecfd8e6554f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:46.049563 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:46.049523 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4d08194-c737-4a26-9198-ecfd8e6554f0-tls-certs" pod "a4d08194-c737-4a26-9198-ecfd8e6554f0" (UID: "a4d08194-c737-4a26-9198-ecfd8e6554f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:05:46.050346 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:46.050280 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d08194-c737-4a26-9198-ecfd8e6554f0-dshm" pod "a4d08194-c737-4a26-9198-ecfd8e6554f0" (UID: "a4d08194-c737-4a26-9198-ecfd8e6554f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:46.050840 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:46.050798 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4d08194-c737-4a26-9198-ecfd8e6554f0-kube-api-access-ctdlh" pod "a4d08194-c737-4a26-9198-ecfd8e6554f0" (UID: "a4d08194-c737-4a26-9198-ecfd8e6554f0"). InnerVolumeSpecName "kube-api-access-ctdlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:05:46.150029 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:46.150001 4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4d08194-c737-4a26-9198-ecfd8e6554f0-dshm\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:46.150029 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:46.150031 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctdlh\" (UniqueName: \"kubernetes.io/projected/a4d08194-c737-4a26-9198-ecfd8e6554f0-kube-api-access-ctdlh\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:46.150219 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:46.150043 4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4d08194-c737-4a26-9198-ecfd8e6554f0-tls-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:46.150219 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:46.150052 4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4d08194-c737-4a26-9198-ecfd8e6554f0-tmp-dir\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:46.552285 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:46.552241 4471 generic.go:356] "Generic (PLEG): container finished" podID="a4d08194-c737-4a26-9198-ecfd8e6554f0" containerID="89cd0366ba9b02dc7892197fca2c1426b66ef294d90b91a736cffb13bc3932ff" exitCode=0 Nov 02 18:05:46.552488 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:46.552290 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm" Nov 02 18:05:46.552488 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:46.552352 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm" event={"ID":"a4d08194-c737-4a26-9198-ecfd8e6554f0","Type":"ContainerDied","Data":"89cd0366ba9b02dc7892197fca2c1426b66ef294d90b91a736cffb13bc3932ff"} Nov 02 18:05:46.552488 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:46.552393 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm" event={"ID":"a4d08194-c737-4a26-9198-ecfd8e6554f0","Type":"ContainerDied","Data":"603b36bfb44d77c8146a2fdeac837d77ed75ac0b254f485a012d43b50d2b6272"} Nov 02 18:05:46.552488 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:46.552409 4471 scope.go:122] "RemoveContainer" containerID="89cd0366ba9b02dc7892197fca2c1426b66ef294d90b91a736cffb13bc3932ff" Nov 02 18:05:46.575241 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:46.575219 4471 scope.go:122] "RemoveContainer" containerID="89cd0366ba9b02dc7892197fca2c1426b66ef294d90b91a736cffb13bc3932ff" Nov 02 18:05:46.575553 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:05:46.575533 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89cd0366ba9b02dc7892197fca2c1426b66ef294d90b91a736cffb13bc3932ff\": container with ID starting with 89cd0366ba9b02dc7892197fca2c1426b66ef294d90b91a736cffb13bc3932ff not found: ID does not exist" containerID="89cd0366ba9b02dc7892197fca2c1426b66ef294d90b91a736cffb13bc3932ff" Nov 02 18:05:46.575646 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:46.575561 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89cd0366ba9b02dc7892197fca2c1426b66ef294d90b91a736cffb13bc3932ff"} err="failed to get container status \"89cd0366ba9b02dc7892197fca2c1426b66ef294d90b91a736cffb13bc3932ff\": rpc error: code = NotFound desc = could not find container \"89cd0366ba9b02dc7892197fca2c1426b66ef294d90b91a736cffb13bc3932ff\": container with ID starting with 89cd0366ba9b02dc7892197fca2c1426b66ef294d90b91a736cffb13bc3932ff not found: ID does not exist" Nov 02 18:05:46.611289 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:46.611248 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm"] Nov 02 18:05:46.623708 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:46.623684 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a29999/test-model-a29999-kserve-768f76f89c-67lgm"] Nov 02 18:05:46.979239 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:46.979201 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" podUID="c3e6eb3c-0496-44c8-a45f-9b4c662756b1" containerName="main" containerID="cri-o://6007b74909a21a6e1b4eb5c94fd5a6539e8493b3dc0daef0a807dc9799c0a5e5" gracePeriod=45 Nov 02 18:05:46.991913 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:46.991875 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" podUID="c3e6eb3c-0496-44c8-a45f-9b4c662756b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.155:8000/ready\": dial tcp 10.133.0.155:8000: connect: connection refused" Nov 02 18:05:47.297740 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.297709 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" Nov 02 18:05:47.383586 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.383552 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-tmp-dir\") pod \"c3e6eb3c-0496-44c8-a45f-9b4c662756b1\" (UID: \"c3e6eb3c-0496-44c8-a45f-9b4c662756b1\") " Nov 02 18:05:47.383798 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.383701 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-dshm\") pod \"c3e6eb3c-0496-44c8-a45f-9b4c662756b1\" (UID: \"c3e6eb3c-0496-44c8-a45f-9b4c662756b1\") " Nov 02 18:05:47.383890 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.383862 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-tmp-dir" pod "c3e6eb3c-0496-44c8-a45f-9b4c662756b1" (UID: "c3e6eb3c-0496-44c8-a45f-9b4c662756b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:47.383890 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.383881 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-kube-api-access-bqflw\" (UniqueName: \"kubernetes.io/projected/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-kube-api-access-bqflw\") pod \"c3e6eb3c-0496-44c8-a45f-9b4c662756b1\" (UID: \"c3e6eb3c-0496-44c8-a45f-9b4c662756b1\") " Nov 02 18:05:47.383991 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.383906 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-model-cache\") pod \"c3e6eb3c-0496-44c8-a45f-9b4c662756b1\" (UID: \"c3e6eb3c-0496-44c8-a45f-9b4c662756b1\") " Nov 02 18:05:47.384040 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.384026 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-tls-certs\" (UniqueName: \"kubernetes.io/secret/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-tls-certs\") pod \"c3e6eb3c-0496-44c8-a45f-9b4c662756b1\" (UID: \"c3e6eb3c-0496-44c8-a45f-9b4c662756b1\") " Nov 02 18:05:47.384156 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.384141 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-home\" (UniqueName: \"kubernetes.io/empty-dir/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-home\") pod \"c3e6eb3c-0496-44c8-a45f-9b4c662756b1\" (UID: \"c3e6eb3c-0496-44c8-a45f-9b4c662756b1\") " Nov 02 18:05:47.384351 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.384301 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-model-cache" pod "c3e6eb3c-0496-44c8-a45f-9b4c662756b1" (UID: "c3e6eb3c-0496-44c8-a45f-9b4c662756b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:47.384508 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.384447 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-home" pod "c3e6eb3c-0496-44c8-a45f-9b4c662756b1" (UID: "c3e6eb3c-0496-44c8-a45f-9b4c662756b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:47.384929 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.384906 4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-model-cache\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:47.385028 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.384934 4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-home\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:47.385028 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.384951 4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-tmp-dir\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:47.386454 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.386427 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-dshm" pod "c3e6eb3c-0496-44c8-a45f-9b4c662756b1" (UID: "c3e6eb3c-0496-44c8-a45f-9b4c662756b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:47.386801 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.386771 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-kube-api-access-bqflw" pod "c3e6eb3c-0496-44c8-a45f-9b4c662756b1" (UID: "c3e6eb3c-0496-44c8-a45f-9b4c662756b1"). InnerVolumeSpecName "kube-api-access-bqflw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:05:47.386893 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.386845 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-tls-certs" pod "c3e6eb3c-0496-44c8-a45f-9b4c662756b1" (UID: "c3e6eb3c-0496-44c8-a45f-9b4c662756b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:05:47.487109 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.487079 4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-dshm\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:47.487109 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.487104 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqflw\" (UniqueName: \"kubernetes.io/projected/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-kube-api-access-bqflw\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:47.487109 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.487115 4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3e6eb3c-0496-44c8-a45f-9b4c662756b1-tls-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:47.563566 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.563538 4471 generic.go:356] "Generic (PLEG): container finished" podID="c3e6eb3c-0496-44c8-a45f-9b4c662756b1" containerID="6007b74909a21a6e1b4eb5c94fd5a6539e8493b3dc0daef0a807dc9799c0a5e5" exitCode=0 Nov 02 18:05:47.563566 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.563570 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" event={"ID":"c3e6eb3c-0496-44c8-a45f-9b4c662756b1","Type":"ContainerDied","Data":"6007b74909a21a6e1b4eb5c94fd5a6539e8493b3dc0daef0a807dc9799c0a5e5"} Nov 02 18:05:47.563798 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.563578 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" Nov 02 18:05:47.563798 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.563597 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5" event={"ID":"c3e6eb3c-0496-44c8-a45f-9b4c662756b1","Type":"ContainerDied","Data":"cc31a17d9d429dc0632539ee49868c76d5820de016ea82b19ace4c1752b8ff69"} Nov 02 18:05:47.563798 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.563614 4471 scope.go:122] "RemoveContainer" containerID="6007b74909a21a6e1b4eb5c94fd5a6539e8493b3dc0daef0a807dc9799c0a5e5" Nov 02 18:05:47.564294 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.564254 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" podUID="fad2db08-e940-4823-99ee-9b3fb1346140" containerName="main" containerID="cri-o://cd93bcb19130cbbe6b61530213047ebdc8dd4ac371b9b9485bc6a8c12b362291" gracePeriod=45 Nov 02 18:05:47.576791 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.576771 4471 scope.go:122] "RemoveContainer" containerID="6007b74909a21a6e1b4eb5c94fd5a6539e8493b3dc0daef0a807dc9799c0a5e5" Nov 02 18:05:47.577055 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:05:47.577029 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6007b74909a21a6e1b4eb5c94fd5a6539e8493b3dc0daef0a807dc9799c0a5e5\": container with ID starting with 6007b74909a21a6e1b4eb5c94fd5a6539e8493b3dc0daef0a807dc9799c0a5e5 not found: ID does not exist" containerID="6007b74909a21a6e1b4eb5c94fd5a6539e8493b3dc0daef0a807dc9799c0a5e5" Nov 02 18:05:47.577124 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.577065 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6007b74909a21a6e1b4eb5c94fd5a6539e8493b3dc0daef0a807dc9799c0a5e5"} err="failed to get container status \"6007b74909a21a6e1b4eb5c94fd5a6539e8493b3dc0daef0a807dc9799c0a5e5\": rpc error: code = NotFound desc = could not find container \"6007b74909a21a6e1b4eb5c94fd5a6539e8493b3dc0daef0a807dc9799c0a5e5\": container with ID starting with 6007b74909a21a6e1b4eb5c94fd5a6539e8493b3dc0daef0a807dc9799c0a5e5 not found: ID does not exist" Nov 02 18:05:47.620400 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.620371 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5"] Nov 02 18:05:47.625205 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.625182 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-bb6d2e/test-model-bb6d2e-kserve-6d74447d6f-9wrh5"] Nov 02 18:05:47.848791 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.848765 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" Nov 02 18:05:47.893266 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.893234 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fad2db08-e940-4823-99ee-9b3fb1346140-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fad2db08-e940-4823-99ee-9b3fb1346140-model-cache\") pod \"fad2db08-e940-4823-99ee-9b3fb1346140\" (UID: \"fad2db08-e940-4823-99ee-9b3fb1346140\") " Nov 02 18:05:47.893491 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.893383 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fad2db08-e940-4823-99ee-9b3fb1346140-tls-certs\" (UniqueName: \"kubernetes.io/secret/fad2db08-e940-4823-99ee-9b3fb1346140-tls-certs\") pod \"fad2db08-e940-4823-99ee-9b3fb1346140\" (UID: \"fad2db08-e940-4823-99ee-9b3fb1346140\") " Nov 02 18:05:47.893583 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.893536 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fad2db08-e940-4823-99ee-9b3fb1346140-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fad2db08-e940-4823-99ee-9b3fb1346140-tmp-dir\") pod \"fad2db08-e940-4823-99ee-9b3fb1346140\" (UID: \"fad2db08-e940-4823-99ee-9b3fb1346140\") " Nov 02 18:05:47.893647 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.893612 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fad2db08-e940-4823-99ee-9b3fb1346140-model-cache" pod "fad2db08-e940-4823-99ee-9b3fb1346140" (UID: "fad2db08-e940-4823-99ee-9b3fb1346140"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:47.893802 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.893779 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fad2db08-e940-4823-99ee-9b3fb1346140-tmp-dir" pod "fad2db08-e940-4823-99ee-9b3fb1346140" (UID: "fad2db08-e940-4823-99ee-9b3fb1346140"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:47.894001 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.893986 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fad2db08-e940-4823-99ee-9b3fb1346140-home\" (UniqueName: \"kubernetes.io/empty-dir/fad2db08-e940-4823-99ee-9b3fb1346140-home\") pod \"fad2db08-e940-4823-99ee-9b3fb1346140\" (UID: \"fad2db08-e940-4823-99ee-9b3fb1346140\") " Nov 02 18:05:47.894138 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.894121 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fad2db08-e940-4823-99ee-9b3fb1346140-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fad2db08-e940-4823-99ee-9b3fb1346140-dshm\") pod \"fad2db08-e940-4823-99ee-9b3fb1346140\" (UID: \"fad2db08-e940-4823-99ee-9b3fb1346140\") " Nov 02 18:05:47.894656 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.894222 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fad2db08-e940-4823-99ee-9b3fb1346140-kube-api-access-gccq9\" (UniqueName: \"kubernetes.io/projected/fad2db08-e940-4823-99ee-9b3fb1346140-kube-api-access-gccq9\") pod \"fad2db08-e940-4823-99ee-9b3fb1346140\" (UID: \"fad2db08-e940-4823-99ee-9b3fb1346140\") " Nov 02 18:05:47.894656 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.894235 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fad2db08-e940-4823-99ee-9b3fb1346140-home" pod "fad2db08-e940-4823-99ee-9b3fb1346140" (UID: "fad2db08-e940-4823-99ee-9b3fb1346140"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:47.895660 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.895334 4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fad2db08-e940-4823-99ee-9b3fb1346140-home\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:47.895660 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.895362 4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fad2db08-e940-4823-99ee-9b3fb1346140-model-cache\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:47.895660 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.895376 4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fad2db08-e940-4823-99ee-9b3fb1346140-tmp-dir\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:47.896472 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.896439 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fad2db08-e940-4823-99ee-9b3fb1346140-tls-certs" pod "fad2db08-e940-4823-99ee-9b3fb1346140" (UID: "fad2db08-e940-4823-99ee-9b3fb1346140"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:05:47.896608 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.896583 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fad2db08-e940-4823-99ee-9b3fb1346140-kube-api-access-gccq9" pod "fad2db08-e940-4823-99ee-9b3fb1346140" (UID: "fad2db08-e940-4823-99ee-9b3fb1346140"). InnerVolumeSpecName "kube-api-access-gccq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:05:47.897357 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.897341 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fad2db08-e940-4823-99ee-9b3fb1346140-dshm" pod "fad2db08-e940-4823-99ee-9b3fb1346140" (UID: "fad2db08-e940-4823-99ee-9b3fb1346140"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:47.997823 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.997784 4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fad2db08-e940-4823-99ee-9b3fb1346140-tls-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:47.997823 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.997819 4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fad2db08-e940-4823-99ee-9b3fb1346140-dshm\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:47.997823 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:47.997830 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gccq9\" (UniqueName: \"kubernetes.io/projected/fad2db08-e940-4823-99ee-9b3fb1346140-kube-api-access-gccq9\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:48.057934 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.057892 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn" podUID="24485a36-8054-44f1-a2ff-f8471e726249" containerName="main" containerID="cri-o://e4320f149008676e4c5a3c9e9cc691584fe8d302ca40415bfb596b6796d5231f" gracePeriod=45 Nov 02 18:05:48.179381 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.176842 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn" podUID="24485a36-8054-44f1-a2ff-f8471e726249" containerName="main" probeResult="failure" output="Get \"https://10.133.0.182:8000/ready\": dial tcp 10.133.0.182:8000: connect: connection refused" Nov 02 18:05:48.445172 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.445139 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4d08194-c737-4a26-9198-ecfd8e6554f0" path="/var/lib/kubelet/pods/a4d08194-c737-4a26-9198-ecfd8e6554f0/volumes" Nov 02 18:05:48.445847 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.445823 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3e6eb3c-0496-44c8-a45f-9b4c662756b1" path="/var/lib/kubelet/pods/c3e6eb3c-0496-44c8-a45f-9b4c662756b1/volumes" Nov 02 18:05:48.474020 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.473996 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn" Nov 02 18:05:48.513853 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.513824 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24485a36-8054-44f1-a2ff-f8471e726249-tls-certs\" (UniqueName: \"kubernetes.io/secret/24485a36-8054-44f1-a2ff-f8471e726249-tls-certs\") pod \"24485a36-8054-44f1-a2ff-f8471e726249\" (UID: \"24485a36-8054-44f1-a2ff-f8471e726249\") " Nov 02 18:05:48.514020 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.513878 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24485a36-8054-44f1-a2ff-f8471e726249-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24485a36-8054-44f1-a2ff-f8471e726249-model-cache\") pod \"24485a36-8054-44f1-a2ff-f8471e726249\" (UID: \"24485a36-8054-44f1-a2ff-f8471e726249\") " Nov 02 18:05:48.514087 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.514032 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24485a36-8054-44f1-a2ff-f8471e726249-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24485a36-8054-44f1-a2ff-f8471e726249-tmp-dir\") pod \"24485a36-8054-44f1-a2ff-f8471e726249\" (UID: \"24485a36-8054-44f1-a2ff-f8471e726249\") " Nov 02 18:05:48.514138 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.514108 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24485a36-8054-44f1-a2ff-f8471e726249-home\" (UniqueName: \"kubernetes.io/empty-dir/24485a36-8054-44f1-a2ff-f8471e726249-home\") pod \"24485a36-8054-44f1-a2ff-f8471e726249\" (UID: \"24485a36-8054-44f1-a2ff-f8471e726249\") " Nov 02 18:05:48.514186 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.514169 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24485a36-8054-44f1-a2ff-f8471e726249-dshm\" (UniqueName: \"kubernetes.io/empty-dir/24485a36-8054-44f1-a2ff-f8471e726249-dshm\") pod \"24485a36-8054-44f1-a2ff-f8471e726249\" (UID: \"24485a36-8054-44f1-a2ff-f8471e726249\") " Nov 02 18:05:48.514234 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.514188 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24485a36-8054-44f1-a2ff-f8471e726249-model-cache" pod "24485a36-8054-44f1-a2ff-f8471e726249" (UID: "24485a36-8054-44f1-a2ff-f8471e726249"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:48.514348 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.514290 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24485a36-8054-44f1-a2ff-f8471e726249-tmp-dir" pod "24485a36-8054-44f1-a2ff-f8471e726249" (UID: "24485a36-8054-44f1-a2ff-f8471e726249"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:48.514418 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.514362 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24485a36-8054-44f1-a2ff-f8471e726249-home" pod "24485a36-8054-44f1-a2ff-f8471e726249" (UID: "24485a36-8054-44f1-a2ff-f8471e726249"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:48.514418 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.514393 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24485a36-8054-44f1-a2ff-f8471e726249-kube-api-access-jfvbg\" (UniqueName: \"kubernetes.io/projected/24485a36-8054-44f1-a2ff-f8471e726249-kube-api-access-jfvbg\") pod \"24485a36-8054-44f1-a2ff-f8471e726249\" (UID: \"24485a36-8054-44f1-a2ff-f8471e726249\") " Nov 02 18:05:48.515219 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.515195 4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24485a36-8054-44f1-a2ff-f8471e726249-tmp-dir\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:48.515219 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.515217 4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24485a36-8054-44f1-a2ff-f8471e726249-home\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:48.515467 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.515226 4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24485a36-8054-44f1-a2ff-f8471e726249-model-cache\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:48.517728 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.517698 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24485a36-8054-44f1-a2ff-f8471e726249-kube-api-access-jfvbg" pod "24485a36-8054-44f1-a2ff-f8471e726249" (UID: "24485a36-8054-44f1-a2ff-f8471e726249"). InnerVolumeSpecName "kube-api-access-jfvbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:05:48.518223 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.518194 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24485a36-8054-44f1-a2ff-f8471e726249-tls-certs" pod "24485a36-8054-44f1-a2ff-f8471e726249" (UID: "24485a36-8054-44f1-a2ff-f8471e726249"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:05:48.519430 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.519403 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24485a36-8054-44f1-a2ff-f8471e726249-dshm" pod "24485a36-8054-44f1-a2ff-f8471e726249" (UID: "24485a36-8054-44f1-a2ff-f8471e726249"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:48.581402 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.581362 4471 generic.go:356] "Generic (PLEG): container finished" podID="24485a36-8054-44f1-a2ff-f8471e726249" containerID="e4320f149008676e4c5a3c9e9cc691584fe8d302ca40415bfb596b6796d5231f" exitCode=0 Nov 02 18:05:48.581612 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.581420 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn" Nov 02 18:05:48.581612 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.581460 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn" event={"ID":"24485a36-8054-44f1-a2ff-f8471e726249","Type":"ContainerDied","Data":"e4320f149008676e4c5a3c9e9cc691584fe8d302ca40415bfb596b6796d5231f"} Nov 02 18:05:48.581612 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.581519 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn" event={"ID":"24485a36-8054-44f1-a2ff-f8471e726249","Type":"ContainerDied","Data":"d4850b1964293d5356ba6d6378fb5632c7887a71cc24200a2a29856555d834d3"} Nov 02 18:05:48.581612 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.581552 4471 scope.go:122] "RemoveContainer" containerID="e4320f149008676e4c5a3c9e9cc691584fe8d302ca40415bfb596b6796d5231f" Nov 02 18:05:48.592354 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.590548 4471 generic.go:356] "Generic (PLEG): container finished" podID="fad2db08-e940-4823-99ee-9b3fb1346140" containerID="cd93bcb19130cbbe6b61530213047ebdc8dd4ac371b9b9485bc6a8c12b362291" exitCode=0 Nov 02 18:05:48.592354 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.590648 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" Nov 02 18:05:48.592354 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.590677 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" event={"ID":"fad2db08-e940-4823-99ee-9b3fb1346140","Type":"ContainerDied","Data":"cd93bcb19130cbbe6b61530213047ebdc8dd4ac371b9b9485bc6a8c12b362291"} Nov 02 18:05:48.592354 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.590704 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c" event={"ID":"fad2db08-e940-4823-99ee-9b3fb1346140","Type":"ContainerDied","Data":"7777675afb8acd43cea6ef25c90a1d256c86a21ad91cc4e6a82b38458942a78a"} Nov 02 18:05:48.596266 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.596242 4471 scope.go:122] "RemoveContainer" containerID="e4320f149008676e4c5a3c9e9cc691584fe8d302ca40415bfb596b6796d5231f" Nov 02 18:05:48.596683 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:05:48.596659 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4320f149008676e4c5a3c9e9cc691584fe8d302ca40415bfb596b6796d5231f\": container with ID starting with e4320f149008676e4c5a3c9e9cc691584fe8d302ca40415bfb596b6796d5231f not found: ID does not exist" containerID="e4320f149008676e4c5a3c9e9cc691584fe8d302ca40415bfb596b6796d5231f" Nov 02 18:05:48.596756 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.596691 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4320f149008676e4c5a3c9e9cc691584fe8d302ca40415bfb596b6796d5231f"} err="failed to get container status \"e4320f149008676e4c5a3c9e9cc691584fe8d302ca40415bfb596b6796d5231f\": rpc error: code = NotFound desc = could not find container \"e4320f149008676e4c5a3c9e9cc691584fe8d302ca40415bfb596b6796d5231f\": container with ID starting with e4320f149008676e4c5a3c9e9cc691584fe8d302ca40415bfb596b6796d5231f not found: ID does not exist" Nov 02 18:05:48.596756 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.596712 4471 scope.go:122] "RemoveContainer" containerID="cd93bcb19130cbbe6b61530213047ebdc8dd4ac371b9b9485bc6a8c12b362291" Nov 02 18:05:48.607002 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.606984 4471 scope.go:122] "RemoveContainer" containerID="cd93bcb19130cbbe6b61530213047ebdc8dd4ac371b9b9485bc6a8c12b362291" Nov 02 18:05:48.607290 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:05:48.607270 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd93bcb19130cbbe6b61530213047ebdc8dd4ac371b9b9485bc6a8c12b362291\": container with ID starting with cd93bcb19130cbbe6b61530213047ebdc8dd4ac371b9b9485bc6a8c12b362291 not found: ID does not exist" containerID="cd93bcb19130cbbe6b61530213047ebdc8dd4ac371b9b9485bc6a8c12b362291" Nov 02 18:05:48.607365 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.607297 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd93bcb19130cbbe6b61530213047ebdc8dd4ac371b9b9485bc6a8c12b362291"} err="failed to get container status \"cd93bcb19130cbbe6b61530213047ebdc8dd4ac371b9b9485bc6a8c12b362291\": rpc error: code = NotFound desc = could not find container \"cd93bcb19130cbbe6b61530213047ebdc8dd4ac371b9b9485bc6a8c12b362291\": container with ID starting with cd93bcb19130cbbe6b61530213047ebdc8dd4ac371b9b9485bc6a8c12b362291 not found: ID does not exist" Nov 02 18:05:48.616531 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.616506 4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24485a36-8054-44f1-a2ff-f8471e726249-tls-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:48.616531 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.616531 4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24485a36-8054-44f1-a2ff-f8471e726249-dshm\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:48.616666 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.616543 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfvbg\" (UniqueName: \"kubernetes.io/projected/24485a36-8054-44f1-a2ff-f8471e726249-kube-api-access-jfvbg\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:48.639300 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.639272 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn"] Nov 02 18:05:48.649611 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.649564 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-872354/test-model-872354-kserve-8f6cd8fd9-xwvbn"] Nov 02 18:05:48.678841 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.678799 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c"] Nov 02 18:05:48.684249 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:48.684217 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-96768b/test-model-96768b-kserve-74d57ffdc-l9t5c"] Nov 02 18:05:50.446866 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:50.446833 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24485a36-8054-44f1-a2ff-f8471e726249" path="/var/lib/kubelet/pods/24485a36-8054-44f1-a2ff-f8471e726249/volumes" Nov 02 18:05:50.447593 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:50.447574 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fad2db08-e940-4823-99ee-9b3fb1346140" path="/var/lib/kubelet/pods/fad2db08-e940-4823-99ee-9b3fb1346140/volumes" Nov 02 18:05:54.298066 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:54.297977 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx" podUID="9b1845b8-af82-4fd6-8950-cfccf0c9ea8e" containerName="main" containerID="cri-o://16a29e98549e5d0817ffa7047de72da3177ef7efcad7d57b45579a0d515c51c0" gracePeriod=45 Nov 02 18:05:54.611252 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:54.611226 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx" Nov 02 18:05:54.645631 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:54.645598 4471 generic.go:356] "Generic (PLEG): container finished" podID="9b1845b8-af82-4fd6-8950-cfccf0c9ea8e" containerID="16a29e98549e5d0817ffa7047de72da3177ef7efcad7d57b45579a0d515c51c0" exitCode=0 Nov 02 18:05:54.645788 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:54.645639 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx" Nov 02 18:05:54.645788 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:54.645681 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx" event={"ID":"9b1845b8-af82-4fd6-8950-cfccf0c9ea8e","Type":"ContainerDied","Data":"16a29e98549e5d0817ffa7047de72da3177ef7efcad7d57b45579a0d515c51c0"} Nov 02 18:05:54.645788 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:54.645726 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx" event={"ID":"9b1845b8-af82-4fd6-8950-cfccf0c9ea8e","Type":"ContainerDied","Data":"31fb05635e85fec480ae47f4d31be684a860a1dad143e696c06672858d4363ea"} Nov 02 18:05:54.645788 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:54.645747 4471 scope.go:122] "RemoveContainer" containerID="16a29e98549e5d0817ffa7047de72da3177ef7efcad7d57b45579a0d515c51c0" Nov 02 18:05:54.657191 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:54.657168 4471 scope.go:122] "RemoveContainer" containerID="16a29e98549e5d0817ffa7047de72da3177ef7efcad7d57b45579a0d515c51c0" Nov 02 18:05:54.657566 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:05:54.657532 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16a29e98549e5d0817ffa7047de72da3177ef7efcad7d57b45579a0d515c51c0\": container with ID starting with 16a29e98549e5d0817ffa7047de72da3177ef7efcad7d57b45579a0d515c51c0 not found: ID does not exist" containerID="16a29e98549e5d0817ffa7047de72da3177ef7efcad7d57b45579a0d515c51c0" Nov 02 18:05:54.657737 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:54.657704 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16a29e98549e5d0817ffa7047de72da3177ef7efcad7d57b45579a0d515c51c0"} err="failed to get container status \"16a29e98549e5d0817ffa7047de72da3177ef7efcad7d57b45579a0d515c51c0\": rpc error: code = NotFound desc = could not find container \"16a29e98549e5d0817ffa7047de72da3177ef7efcad7d57b45579a0d515c51c0\": container with ID starting with 16a29e98549e5d0817ffa7047de72da3177ef7efcad7d57b45579a0d515c51c0 not found: ID does not exist" Nov 02 18:05:54.738713 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:54.738680 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-kube-api-access-2b6xq\" (UniqueName: \"kubernetes.io/projected/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-kube-api-access-2b6xq\") pod \"9b1845b8-af82-4fd6-8950-cfccf0c9ea8e\" (UID: \"9b1845b8-af82-4fd6-8950-cfccf0c9ea8e\") " Nov 02 18:05:54.738904 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:54.738749 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-home\" (UniqueName: \"kubernetes.io/empty-dir/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-home\") pod \"9b1845b8-af82-4fd6-8950-cfccf0c9ea8e\" (UID: \"9b1845b8-af82-4fd6-8950-cfccf0c9ea8e\") " Nov 02 18:05:54.738904 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:54.738846 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-tmp-dir\") pod \"9b1845b8-af82-4fd6-8950-cfccf0c9ea8e\" (UID: \"9b1845b8-af82-4fd6-8950-cfccf0c9ea8e\") " Nov 02 18:05:54.739029 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:54.739005 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-home" pod "9b1845b8-af82-4fd6-8950-cfccf0c9ea8e" (UID: "9b1845b8-af82-4fd6-8950-cfccf0c9ea8e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:54.739029 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:54.739019 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-model-cache\") pod \"9b1845b8-af82-4fd6-8950-cfccf0c9ea8e\" (UID: \"9b1845b8-af82-4fd6-8950-cfccf0c9ea8e\") " Nov 02 18:05:54.739154 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:54.739110 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-tmp-dir" pod "9b1845b8-af82-4fd6-8950-cfccf0c9ea8e" (UID: "9b1845b8-af82-4fd6-8950-cfccf0c9ea8e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:54.739235 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:54.739169 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-tls-certs\" (UniqueName: \"kubernetes.io/secret/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-tls-certs\") pod \"9b1845b8-af82-4fd6-8950-cfccf0c9ea8e\" (UID: \"9b1845b8-af82-4fd6-8950-cfccf0c9ea8e\") " Nov 02 18:05:54.739276 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:54.739243 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-model-cache" pod "9b1845b8-af82-4fd6-8950-cfccf0c9ea8e" (UID: "9b1845b8-af82-4fd6-8950-cfccf0c9ea8e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:54.739340 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:54.739317 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-dshm\") pod \"9b1845b8-af82-4fd6-8950-cfccf0c9ea8e\" (UID: \"9b1845b8-af82-4fd6-8950-cfccf0c9ea8e\") " Nov 02 18:05:54.740099 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:54.740075 4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-home\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:54.740099 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:54.740094 4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-tmp-dir\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:54.740099 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:54.740103 4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-model-cache\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:54.741126 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:54.741101 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-kube-api-access-2b6xq" pod "9b1845b8-af82-4fd6-8950-cfccf0c9ea8e" (UID: "9b1845b8-af82-4fd6-8950-cfccf0c9ea8e"). InnerVolumeSpecName "kube-api-access-2b6xq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:05:54.741392 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:54.741369 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-dshm" pod "9b1845b8-af82-4fd6-8950-cfccf0c9ea8e" (UID: "9b1845b8-af82-4fd6-8950-cfccf0c9ea8e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:54.741465 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:54.741397 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-tls-certs" pod "9b1845b8-af82-4fd6-8950-cfccf0c9ea8e" (UID: "9b1845b8-af82-4fd6-8950-cfccf0c9ea8e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:05:54.817699 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:54.817649 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw" podUID="af93e51c-6948-43fd-b8a3-c7fd0c730dc6" containerName="main" containerID="cri-o://88ec3ce0aaef54c4c198e3fdef0834ecceb0d07e86087305d1ec2d728b02e02e" gracePeriod=45 Nov 02 18:05:54.841946 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:54.841920 4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-tls-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:54.841946 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:54.841946 4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-dshm\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:54.842100 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:54.841958 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b6xq\" (UniqueName: \"kubernetes.io/projected/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e-kube-api-access-2b6xq\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:54.998711 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:54.998680 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx"] Nov 02 18:05:55.009365 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.006427 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-5fd3e0/test-model-5fd3e0-kserve-56cc48b75c-dptkx"] Nov 02 18:05:55.101746 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.101721 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw" Nov 02 18:05:55.249409 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.249377 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-tmp-dir\") pod \"af93e51c-6948-43fd-b8a3-c7fd0c730dc6\" (UID: \"af93e51c-6948-43fd-b8a3-c7fd0c730dc6\") " Nov 02 18:05:55.249606 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.249515 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-model-cache\") pod \"af93e51c-6948-43fd-b8a3-c7fd0c730dc6\" (UID: \"af93e51c-6948-43fd-b8a3-c7fd0c730dc6\") " Nov 02 18:05:55.249672 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.249613 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-kube-api-access-67kmv\" (UniqueName: \"kubernetes.io/projected/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-kube-api-access-67kmv\") pod \"af93e51c-6948-43fd-b8a3-c7fd0c730dc6\" (UID: \"af93e51c-6948-43fd-b8a3-c7fd0c730dc6\") " Nov 02 18:05:55.249731 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.249661 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-tmp-dir" pod "af93e51c-6948-43fd-b8a3-c7fd0c730dc6" (UID: "af93e51c-6948-43fd-b8a3-c7fd0c730dc6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:55.249731 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.249673 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-home\" (UniqueName: \"kubernetes.io/empty-dir/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-home\") pod \"af93e51c-6948-43fd-b8a3-c7fd0c730dc6\" (UID: \"af93e51c-6948-43fd-b8a3-c7fd0c730dc6\") " Nov 02 18:05:55.249838 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.249746 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-model-cache" pod "af93e51c-6948-43fd-b8a3-c7fd0c730dc6" (UID: "af93e51c-6948-43fd-b8a3-c7fd0c730dc6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:55.249838 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.249772 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-dshm\") pod \"af93e51c-6948-43fd-b8a3-c7fd0c730dc6\" (UID: \"af93e51c-6948-43fd-b8a3-c7fd0c730dc6\") " Nov 02 18:05:55.249838 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.249795 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-tls-certs\" (UniqueName: \"kubernetes.io/secret/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-tls-certs\") pod \"af93e51c-6948-43fd-b8a3-c7fd0c730dc6\" (UID: \"af93e51c-6948-43fd-b8a3-c7fd0c730dc6\") " Nov 02 18:05:55.250055 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.249887 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-home" pod "af93e51c-6948-43fd-b8a3-c7fd0c730dc6" (UID: "af93e51c-6948-43fd-b8a3-c7fd0c730dc6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:55.250857 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.250834 4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-model-cache\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:55.250956 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.250865 4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-home\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:55.250956 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.250881 4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-tmp-dir\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:55.252033 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.252004 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-dshm" pod "af93e51c-6948-43fd-b8a3-c7fd0c730dc6" (UID: "af93e51c-6948-43fd-b8a3-c7fd0c730dc6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:55.252033 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.252008 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-tls-certs" pod "af93e51c-6948-43fd-b8a3-c7fd0c730dc6" (UID: "af93e51c-6948-43fd-b8a3-c7fd0c730dc6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:05:55.252179 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.252043 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-kube-api-access-67kmv" pod "af93e51c-6948-43fd-b8a3-c7fd0c730dc6" (UID: "af93e51c-6948-43fd-b8a3-c7fd0c730dc6"). InnerVolumeSpecName "kube-api-access-67kmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:05:55.307595 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.307542 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-57d5879ccd-4wcjv" podUID="b536a1cd-25ed-431a-af71-3cb8a0628c67" containerName="console" containerID="cri-o://dddbb32030f6c129993496fbaddb4977ba29089b15f30eb9f898178cf416565a" gracePeriod=15 Nov 02 18:05:55.352327 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.352287 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67kmv\" (UniqueName: \"kubernetes.io/projected/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-kube-api-access-67kmv\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:55.352424 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.352331 4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-dshm\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:55.352424 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.352343 4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af93e51c-6948-43fd-b8a3-c7fd0c730dc6-tls-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:55.609337 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.609246 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d5879ccd-4wcjv" Nov 02 18:05:55.657854 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.657823 4471 generic.go:356] "Generic (PLEG): container finished" podID="af93e51c-6948-43fd-b8a3-c7fd0c730dc6" containerID="88ec3ce0aaef54c4c198e3fdef0834ecceb0d07e86087305d1ec2d728b02e02e" exitCode=0 Nov 02 18:05:55.658018 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.657909 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw" Nov 02 18:05:55.658138 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.658116 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw" event={"ID":"af93e51c-6948-43fd-b8a3-c7fd0c730dc6","Type":"ContainerDied","Data":"88ec3ce0aaef54c4c198e3fdef0834ecceb0d07e86087305d1ec2d728b02e02e"} Nov 02 18:05:55.658228 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.658159 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw" event={"ID":"af93e51c-6948-43fd-b8a3-c7fd0c730dc6","Type":"ContainerDied","Data":"e0171b7f1bdbb183059fd697b101a93b3d0f34cdd49106643d8c2c161e044663"} Nov 02 18:05:55.658228 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.658183 4471 scope.go:122] "RemoveContainer" containerID="88ec3ce0aaef54c4c198e3fdef0834ecceb0d07e86087305d1ec2d728b02e02e" Nov 02 18:05:55.659996 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.659969 4471 generic.go:356] "Generic (PLEG): container finished" podID="b536a1cd-25ed-431a-af71-3cb8a0628c67" containerID="dddbb32030f6c129993496fbaddb4977ba29089b15f30eb9f898178cf416565a" exitCode=0 Nov 02 18:05:55.660113 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.660011 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d5879ccd-4wcjv" Nov 02 18:05:55.660113 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.660050 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d5879ccd-4wcjv" event={"ID":"b536a1cd-25ed-431a-af71-3cb8a0628c67","Type":"ContainerDied","Data":"dddbb32030f6c129993496fbaddb4977ba29089b15f30eb9f898178cf416565a"} Nov 02 18:05:55.660113 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.660110 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d5879ccd-4wcjv" event={"ID":"b536a1cd-25ed-431a-af71-3cb8a0628c67","Type":"ContainerDied","Data":"351a5d6c85c885704aa0fac71e7213ce58a31fddb200fd21c5186d5d6944acf0"} Nov 02 18:05:55.706891 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.703991 4471 scope.go:122] "RemoveContainer" containerID="88ec3ce0aaef54c4c198e3fdef0834ecceb0d07e86087305d1ec2d728b02e02e" Nov 02 18:05:55.718336 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:05:55.716888 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88ec3ce0aaef54c4c198e3fdef0834ecceb0d07e86087305d1ec2d728b02e02e\": container with ID starting with 88ec3ce0aaef54c4c198e3fdef0834ecceb0d07e86087305d1ec2d728b02e02e not found: ID does not exist" containerID="88ec3ce0aaef54c4c198e3fdef0834ecceb0d07e86087305d1ec2d728b02e02e" Nov 02 18:05:55.718336 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.716942 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88ec3ce0aaef54c4c198e3fdef0834ecceb0d07e86087305d1ec2d728b02e02e"} err="failed to get container status \"88ec3ce0aaef54c4c198e3fdef0834ecceb0d07e86087305d1ec2d728b02e02e\": rpc error: code = NotFound desc = could not find container \"88ec3ce0aaef54c4c198e3fdef0834ecceb0d07e86087305d1ec2d728b02e02e\": container with ID starting with 88ec3ce0aaef54c4c198e3fdef0834ecceb0d07e86087305d1ec2d728b02e02e not found: ID does not exist" Nov 02 18:05:55.718336 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.716970 4471 scope.go:122] "RemoveContainer" containerID="dddbb32030f6c129993496fbaddb4977ba29089b15f30eb9f898178cf416565a" Nov 02 18:05:55.767087 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.759145 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b536a1cd-25ed-431a-af71-3cb8a0628c67-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b536a1cd-25ed-431a-af71-3cb8a0628c67-tmp\") pod \"b536a1cd-25ed-431a-af71-3cb8a0628c67\" (UID: \"b536a1cd-25ed-431a-af71-3cb8a0628c67\") " Nov 02 18:05:55.767087 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.759228 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b536a1cd-25ed-431a-af71-3cb8a0628c67-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b536a1cd-25ed-431a-af71-3cb8a0628c67-oauth-serving-cert\") pod \"b536a1cd-25ed-431a-af71-3cb8a0628c67\" (UID: \"b536a1cd-25ed-431a-af71-3cb8a0628c67\") " Nov 02 18:05:55.767087 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.760034 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b536a1cd-25ed-431a-af71-3cb8a0628c67-oauth-serving-cert" pod "b536a1cd-25ed-431a-af71-3cb8a0628c67" (UID: "b536a1cd-25ed-431a-af71-3cb8a0628c67"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:05:55.767087 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.760290 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b536a1cd-25ed-431a-af71-3cb8a0628c67-tmp" pod "b536a1cd-25ed-431a-af71-3cb8a0628c67" (UID: "b536a1cd-25ed-431a-af71-3cb8a0628c67"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:55.767087 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.760406 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b536a1cd-25ed-431a-af71-3cb8a0628c67-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b536a1cd-25ed-431a-af71-3cb8a0628c67-console-oauth-config\") pod \"b536a1cd-25ed-431a-af71-3cb8a0628c67\" (UID: \"b536a1cd-25ed-431a-af71-3cb8a0628c67\") " Nov 02 18:05:55.767087 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.760485 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b536a1cd-25ed-431a-af71-3cb8a0628c67-service-ca\" (UniqueName: \"kubernetes.io/configmap/b536a1cd-25ed-431a-af71-3cb8a0628c67-service-ca\") pod \"b536a1cd-25ed-431a-af71-3cb8a0628c67\" (UID: \"b536a1cd-25ed-431a-af71-3cb8a0628c67\") " Nov 02 18:05:55.767087 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.760588 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b536a1cd-25ed-431a-af71-3cb8a0628c67-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b536a1cd-25ed-431a-af71-3cb8a0628c67-console-serving-cert\") pod \"b536a1cd-25ed-431a-af71-3cb8a0628c67\" (UID: \"b536a1cd-25ed-431a-af71-3cb8a0628c67\") " Nov 02 18:05:55.767087 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.760682 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b536a1cd-25ed-431a-af71-3cb8a0628c67-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b536a1cd-25ed-431a-af71-3cb8a0628c67-trusted-ca-bundle\") pod \"b536a1cd-25ed-431a-af71-3cb8a0628c67\" (UID: \"b536a1cd-25ed-431a-af71-3cb8a0628c67\") " Nov 02 18:05:55.767087 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.760719 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b536a1cd-25ed-431a-af71-3cb8a0628c67-console-config\" (UniqueName: \"kubernetes.io/configmap/b536a1cd-25ed-431a-af71-3cb8a0628c67-console-config\") pod \"b536a1cd-25ed-431a-af71-3cb8a0628c67\" (UID: \"b536a1cd-25ed-431a-af71-3cb8a0628c67\") " Nov 02 18:05:55.767087 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.761439 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b536a1cd-25ed-431a-af71-3cb8a0628c67-service-ca" pod "b536a1cd-25ed-431a-af71-3cb8a0628c67" (UID: "b536a1cd-25ed-431a-af71-3cb8a0628c67"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:05:55.767087 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.761761 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b536a1cd-25ed-431a-af71-3cb8a0628c67-console-config" pod "b536a1cd-25ed-431a-af71-3cb8a0628c67" (UID: "b536a1cd-25ed-431a-af71-3cb8a0628c67"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:05:55.767087 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.762623 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b536a1cd-25ed-431a-af71-3cb8a0628c67-trusted-ca-bundle" pod "b536a1cd-25ed-431a-af71-3cb8a0628c67" (UID: "b536a1cd-25ed-431a-af71-3cb8a0628c67"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:05:55.767087 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.760760 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b536a1cd-25ed-431a-af71-3cb8a0628c67-kube-api-access-n4cnd\" (UniqueName: \"kubernetes.io/projected/b536a1cd-25ed-431a-af71-3cb8a0628c67-kube-api-access-n4cnd\") pod \"b536a1cd-25ed-431a-af71-3cb8a0628c67\" (UID: \"b536a1cd-25ed-431a-af71-3cb8a0628c67\") " Nov 02 18:05:55.767087 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.763686 4471 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b536a1cd-25ed-431a-af71-3cb8a0628c67-trusted-ca-bundle\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:55.767087 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.763709 4471 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b536a1cd-25ed-431a-af71-3cb8a0628c67-console-config\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:55.767087 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.763724 4471 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b536a1cd-25ed-431a-af71-3cb8a0628c67-tmp\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:55.767087 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.763737 4471 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b536a1cd-25ed-431a-af71-3cb8a0628c67-oauth-serving-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:55.767087 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.763749 4471 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b536a1cd-25ed-431a-af71-3cb8a0628c67-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:55.830021 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.829041 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b536a1cd-25ed-431a-af71-3cb8a0628c67-console-serving-cert" pod "b536a1cd-25ed-431a-af71-3cb8a0628c67" (UID: "b536a1cd-25ed-431a-af71-3cb8a0628c67"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:05:55.830021 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.829135 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b536a1cd-25ed-431a-af71-3cb8a0628c67-console-oauth-config" pod "b536a1cd-25ed-431a-af71-3cb8a0628c67" (UID: "b536a1cd-25ed-431a-af71-3cb8a0628c67"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:05:55.830021 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.829248 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b536a1cd-25ed-431a-af71-3cb8a0628c67-kube-api-access-n4cnd" pod "b536a1cd-25ed-431a-af71-3cb8a0628c67" (UID: "b536a1cd-25ed-431a-af71-3cb8a0628c67"). InnerVolumeSpecName "kube-api-access-n4cnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:05:55.830674 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.830655 4471 scope.go:122] "RemoveContainer" containerID="dddbb32030f6c129993496fbaddb4977ba29089b15f30eb9f898178cf416565a" Nov 02 18:05:55.831375 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:05:55.831108 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dddbb32030f6c129993496fbaddb4977ba29089b15f30eb9f898178cf416565a\": container with ID starting with dddbb32030f6c129993496fbaddb4977ba29089b15f30eb9f898178cf416565a not found: ID does not exist" containerID="dddbb32030f6c129993496fbaddb4977ba29089b15f30eb9f898178cf416565a" Nov 02 18:05:55.831375 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.831144 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dddbb32030f6c129993496fbaddb4977ba29089b15f30eb9f898178cf416565a"} err="failed to get container status \"dddbb32030f6c129993496fbaddb4977ba29089b15f30eb9f898178cf416565a\": rpc error: code = NotFound desc = could not find container \"dddbb32030f6c129993496fbaddb4977ba29089b15f30eb9f898178cf416565a\": container with ID starting with dddbb32030f6c129993496fbaddb4977ba29089b15f30eb9f898178cf416565a not found: ID does not exist" Nov 02 18:05:55.856478 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.856430 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw"] Nov 02 18:05:55.861508 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.861429 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-fbbbf8/test-model-fbbbf8-kserve-5b4b5dc66c-fmrtw"] Nov 02 18:05:55.865288 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.865266 4471 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b536a1cd-25ed-431a-af71-3cb8a0628c67-console-oauth-config\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:55.865408 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.865290 4471 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b536a1cd-25ed-431a-af71-3cb8a0628c67-console-serving-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:55.865408 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:55.865300 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4cnd\" (UniqueName: \"kubernetes.io/projected/b536a1cd-25ed-431a-af71-3cb8a0628c67-kube-api-access-n4cnd\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:05:56.023260 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:56.023223 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57d5879ccd-4wcjv"] Nov 02 18:05:56.031990 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:56.031875 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57d5879ccd-4wcjv"] Nov 02 18:05:56.444785 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:56.444751 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b1845b8-af82-4fd6-8950-cfccf0c9ea8e" path="/var/lib/kubelet/pods/9b1845b8-af82-4fd6-8950-cfccf0c9ea8e/volumes" Nov 02 18:05:56.445319 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:56.445292 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af93e51c-6948-43fd-b8a3-c7fd0c730dc6" path="/var/lib/kubelet/pods/af93e51c-6948-43fd-b8a3-c7fd0c730dc6/volumes" Nov 02 18:05:56.445822 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:56.445807 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b536a1cd-25ed-431a-af71-3cb8a0628c67" path="/var/lib/kubelet/pods/b536a1cd-25ed-431a-af71-3cb8a0628c67/volumes" Nov 02 18:05:59.108029 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.107996 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q"] Nov 02 18:05:59.108955 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.108936 4471 cpu_manager.go:414] "containerMap: removing container" podUID="24485a36-8054-44f1-a2ff-f8471e726249" containerName="main" Nov 02 18:05:59.109015 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.108956 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24485a36-8054-44f1-a2ff-f8471e726249" containerName="main" Nov 02 18:05:59.109015 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.108975 4471 cpu_manager.go:414] "containerMap: removing container" podUID="c3e6eb3c-0496-44c8-a45f-9b4c662756b1" containerName="main" Nov 02 18:05:59.109015 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.108980 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3e6eb3c-0496-44c8-a45f-9b4c662756b1" containerName="main" Nov 02 18:05:59.109015 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.108994 4471 cpu_manager.go:414] "containerMap: removing container" podUID="fad2db08-e940-4823-99ee-9b3fb1346140" containerName="main" Nov 02 18:05:59.109015 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.109000 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fad2db08-e940-4823-99ee-9b3fb1346140" containerName="main" Nov 02 18:05:59.109180 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.109017 4471 cpu_manager.go:414] "containerMap: removing container" podUID="b536a1cd-25ed-431a-af71-3cb8a0628c67" containerName="console" Nov 02 18:05:59.109180 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.109023 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b536a1cd-25ed-431a-af71-3cb8a0628c67" containerName="console" Nov 02 18:05:59.109180 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.109032 4471 cpu_manager.go:414] "containerMap: removing container" podUID="af93e51c-6948-43fd-b8a3-c7fd0c730dc6" containerName="main" Nov 02 18:05:59.109180 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.109037 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af93e51c-6948-43fd-b8a3-c7fd0c730dc6" containerName="main" Nov 02 18:05:59.109180 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.109051 4471 cpu_manager.go:414] "containerMap: removing container" podUID="9b1845b8-af82-4fd6-8950-cfccf0c9ea8e" containerName="main" Nov 02 18:05:59.109180 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.109055 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b1845b8-af82-4fd6-8950-cfccf0c9ea8e" containerName="main" Nov 02 18:05:59.109180 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.109074 4471 cpu_manager.go:414] "containerMap: removing container" podUID="a4d08194-c737-4a26-9198-ecfd8e6554f0" containerName="main" Nov 02 18:05:59.109180 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.109078 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4d08194-c737-4a26-9198-ecfd8e6554f0" containerName="main" Nov 02 18:05:59.109180 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.109097 4471 cpu_manager.go:414] "containerMap: removing container" podUID="81708d59-e82f-4259-9bf1-cec029c05c73" containerName="istio-proxy" Nov 02 18:05:59.109180 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.109102 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81708d59-e82f-4259-9bf1-cec029c05c73" containerName="istio-proxy" Nov 02 18:05:59.109512 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.109206 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="af93e51c-6948-43fd-b8a3-c7fd0c730dc6" containerName="main" Nov 02 18:05:59.109512 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.109215 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="b536a1cd-25ed-431a-af71-3cb8a0628c67" containerName="console" Nov 02 18:05:59.109512 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.109225 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b1845b8-af82-4fd6-8950-cfccf0c9ea8e" containerName="main" Nov 02 18:05:59.109512 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.109233 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4d08194-c737-4a26-9198-ecfd8e6554f0" containerName="main" Nov 02 18:05:59.109512 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.109243 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="81708d59-e82f-4259-9bf1-cec029c05c73" containerName="istio-proxy" Nov 02 18:05:59.109512 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.109250 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="24485a36-8054-44f1-a2ff-f8471e726249" containerName="main" Nov 02 18:05:59.109512 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.109258 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3e6eb3c-0496-44c8-a45f-9b4c662756b1" containerName="main" Nov 02 18:05:59.109512 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.109266 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="fad2db08-e940-4823-99ee-9b3fb1346140" containerName="main" Nov 02 18:05:59.114916 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.114891 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q" Nov 02 18:05:59.131662 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.131639 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-227939-serving-cert\"" Nov 02 18:05:59.136631 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.136607 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/411e5db0-b883-4372-b0ce-42ced3dd0a9b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q\" (UID: \"411e5db0-b883-4372-b0ce-42ced3dd0a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q" Nov 02 18:05:59.136797 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.136772 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/411e5db0-b883-4372-b0ce-42ced3dd0a9b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q\" (UID: \"411e5db0-b883-4372-b0ce-42ced3dd0a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q" Nov 02 18:05:59.136990 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.136968 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/411e5db0-b883-4372-b0ce-42ced3dd0a9b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q\" (UID: \"411e5db0-b883-4372-b0ce-42ced3dd0a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q" Nov 02 18:05:59.137067 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.137008 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdv4w\" (UniqueName: \"kubernetes.io/projected/411e5db0-b883-4372-b0ce-42ced3dd0a9b-kube-api-access-xdv4w\") pod \"maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q\" (UID: \"411e5db0-b883-4372-b0ce-42ced3dd0a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q" Nov 02 18:05:59.176858 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.176825 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q"] Nov 02 18:05:59.238523 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.238480 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/411e5db0-b883-4372-b0ce-42ced3dd0a9b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q\" (UID: \"411e5db0-b883-4372-b0ce-42ced3dd0a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q" Nov 02 18:05:59.238694 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.238540 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdv4w\" (UniqueName: \"kubernetes.io/projected/411e5db0-b883-4372-b0ce-42ced3dd0a9b-kube-api-access-xdv4w\") pod \"maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q\" (UID: \"411e5db0-b883-4372-b0ce-42ced3dd0a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q" Nov 02 18:05:59.238780 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.238762 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/411e5db0-b883-4372-b0ce-42ced3dd0a9b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q\" (UID: \"411e5db0-b883-4372-b0ce-42ced3dd0a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q" Nov 02 18:05:59.238945 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.238928 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/411e5db0-b883-4372-b0ce-42ced3dd0a9b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q\" (UID: \"411e5db0-b883-4372-b0ce-42ced3dd0a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q" Nov 02 18:05:59.240051 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.240025 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/411e5db0-b883-4372-b0ce-42ced3dd0a9b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q\" (UID: \"411e5db0-b883-4372-b0ce-42ced3dd0a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q" Nov 02 18:05:59.240601 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:05:59.240559 4471 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-227939-serving-cert: secret "maas-api-e2e-shared-b-w3-227939-serving-cert" not found Nov 02 18:05:59.240765 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:05:59.240630 4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/411e5db0-b883-4372-b0ce-42ced3dd0a9b-maas-api-tls podName:411e5db0-b883-4372-b0ce-42ced3dd0a9b nodeName:}" failed. No retries permitted until 2026-11-02 18:05:59.740610764 +0000 UTC m=+2177.435966239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/411e5db0-b883-4372-b0ce-42ced3dd0a9b-maas-api-tls") pod "maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q" (UID: "411e5db0-b883-4372-b0ce-42ced3dd0a9b") : secret "maas-api-e2e-shared-b-w3-227939-serving-cert" not found Nov 02 18:05:59.242556 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.242532 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/411e5db0-b883-4372-b0ce-42ced3dd0a9b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q\" (UID: \"411e5db0-b883-4372-b0ce-42ced3dd0a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q" Nov 02 18:05:59.284041 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.284013 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdv4w\" (UniqueName: \"kubernetes.io/projected/411e5db0-b883-4372-b0ce-42ced3dd0a9b-kube-api-access-xdv4w\") pod \"maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q\" (UID: \"411e5db0-b883-4372-b0ce-42ced3dd0a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q" Nov 02 18:05:59.766635 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.766591 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/411e5db0-b883-4372-b0ce-42ced3dd0a9b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q\" (UID: \"411e5db0-b883-4372-b0ce-42ced3dd0a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q" Nov 02 18:05:59.777358 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:05:59.777269 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/411e5db0-b883-4372-b0ce-42ced3dd0a9b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q\" (UID: \"411e5db0-b883-4372-b0ce-42ced3dd0a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q" Nov 02 18:06:00.030745 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:00.030649 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q" Nov 02 18:06:00.461430 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:06:00.461399 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod411e5db0_b883_4372_b0ce_42ced3dd0a9b.slice/crio-324c3386958002fc3398d2dab194107d1b3b5c86193a1cfc660457112f8b0c7e WatchSource:0}: Error finding container 324c3386958002fc3398d2dab194107d1b3b5c86193a1cfc660457112f8b0c7e: Status 404 returned error can't find the container with id 324c3386958002fc3398d2dab194107d1b3b5c86193a1cfc660457112f8b0c7e Nov 02 18:06:00.464052 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:00.464031 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q"] Nov 02 18:06:00.782325 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:00.781626 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q" event={"ID":"411e5db0-b883-4372-b0ce-42ced3dd0a9b","Type":"ContainerStarted","Data":"324c3386958002fc3398d2dab194107d1b3b5c86193a1cfc660457112f8b0c7e"} Nov 02 18:06:01.791196 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:01.791158 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q" event={"ID":"411e5db0-b883-4372-b0ce-42ced3dd0a9b","Type":"ContainerStarted","Data":"8a337a8562e57563532f886fe2def11cac94cdfbc8649b40b1840ce34cba20b7"} Nov 02 18:06:01.791650 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:01.791264 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q" Nov 02 18:06:01.840292 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:01.840232 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q" podStartSLOduration=2.404799085 podStartE2EDuration="2.840216839s" totalImagesPullingTime="435.417754ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:05:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:06:01.835245215 +0000 UTC m=+2179.530600689" watchObservedRunningTime="2026-11-02 18:06:01.840216839 +0000 UTC m=+2179.535572331" Nov 02 18:06:07.805285 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:07.805245 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q" Nov 02 18:06:09.184993 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:06:09.184951 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:06:09.579289 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:09.579254 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rt5cp_1b5ee10f-87a1-4168-904e-ba233f304fe3/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:15.742167 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:15.742129 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-9b8d5bbf-xnjnp"] Nov 02 18:06:15.749040 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:15.749013 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9b8d5bbf-xnjnp" Nov 02 18:06:15.783714 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:15.783679 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9b8d5bbf-xnjnp"] Nov 02 18:06:15.853557 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:15.853423 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83085553-108f-4d34-b8de-9d0c6e0b13e5-oauth-serving-cert\") pod \"console-9b8d5bbf-xnjnp\" (UID: \"83085553-108f-4d34-b8de-9d0c6e0b13e5\") " pod="openshift-console/console-9b8d5bbf-xnjnp" Nov 02 18:06:15.853557 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:15.853480 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b9p2\" (UniqueName: \"kubernetes.io/projected/83085553-108f-4d34-b8de-9d0c6e0b13e5-kube-api-access-7b9p2\") pod \"console-9b8d5bbf-xnjnp\" (UID: \"83085553-108f-4d34-b8de-9d0c6e0b13e5\") " pod="openshift-console/console-9b8d5bbf-xnjnp" Nov 02 18:06:15.854414 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:15.853972 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83085553-108f-4d34-b8de-9d0c6e0b13e5-trusted-ca-bundle\") pod \"console-9b8d5bbf-xnjnp\" (UID: \"83085553-108f-4d34-b8de-9d0c6e0b13e5\") " pod="openshift-console/console-9b8d5bbf-xnjnp" Nov 02 18:06:15.854414 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:15.854142 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83085553-108f-4d34-b8de-9d0c6e0b13e5-tmp\") pod \"console-9b8d5bbf-xnjnp\" (UID: \"83085553-108f-4d34-b8de-9d0c6e0b13e5\") " pod="openshift-console/console-9b8d5bbf-xnjnp" Nov 02 18:06:15.854414 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:15.854181 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83085553-108f-4d34-b8de-9d0c6e0b13e5-console-serving-cert\") pod \"console-9b8d5bbf-xnjnp\" (UID: \"83085553-108f-4d34-b8de-9d0c6e0b13e5\") " pod="openshift-console/console-9b8d5bbf-xnjnp" Nov 02 18:06:15.854414 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:15.854212 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/83085553-108f-4d34-b8de-9d0c6e0b13e5-console-config\") pod \"console-9b8d5bbf-xnjnp\" (UID: \"83085553-108f-4d34-b8de-9d0c6e0b13e5\") " pod="openshift-console/console-9b8d5bbf-xnjnp" Nov 02 18:06:15.854414 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:15.854375 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/83085553-108f-4d34-b8de-9d0c6e0b13e5-service-ca\") pod \"console-9b8d5bbf-xnjnp\" (UID: \"83085553-108f-4d34-b8de-9d0c6e0b13e5\") " pod="openshift-console/console-9b8d5bbf-xnjnp" Nov 02 18:06:15.854748 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:15.854452 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83085553-108f-4d34-b8de-9d0c6e0b13e5-console-oauth-config\") pod \"console-9b8d5bbf-xnjnp\" (UID: \"83085553-108f-4d34-b8de-9d0c6e0b13e5\") " pod="openshift-console/console-9b8d5bbf-xnjnp" Nov 02 18:06:15.956649 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:15.956615 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83085553-108f-4d34-b8de-9d0c6e0b13e5-trusted-ca-bundle\") pod \"console-9b8d5bbf-xnjnp\" (UID: \"83085553-108f-4d34-b8de-9d0c6e0b13e5\") " pod="openshift-console/console-9b8d5bbf-xnjnp" Nov 02 18:06:15.956967 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:15.956792 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83085553-108f-4d34-b8de-9d0c6e0b13e5-tmp\") pod \"console-9b8d5bbf-xnjnp\" (UID: \"83085553-108f-4d34-b8de-9d0c6e0b13e5\") " pod="openshift-console/console-9b8d5bbf-xnjnp" Nov 02 18:06:15.956967 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:15.956835 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83085553-108f-4d34-b8de-9d0c6e0b13e5-console-serving-cert\") pod \"console-9b8d5bbf-xnjnp\" (UID: \"83085553-108f-4d34-b8de-9d0c6e0b13e5\") " pod="openshift-console/console-9b8d5bbf-xnjnp" Nov 02 18:06:15.956967 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:15.956864 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/83085553-108f-4d34-b8de-9d0c6e0b13e5-console-config\") pod \"console-9b8d5bbf-xnjnp\" (UID: \"83085553-108f-4d34-b8de-9d0c6e0b13e5\") " pod="openshift-console/console-9b8d5bbf-xnjnp" Nov 02 18:06:15.957553 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:15.957016 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/83085553-108f-4d34-b8de-9d0c6e0b13e5-service-ca\") pod \"console-9b8d5bbf-xnjnp\" (UID: \"83085553-108f-4d34-b8de-9d0c6e0b13e5\") " pod="openshift-console/console-9b8d5bbf-xnjnp" Nov 02 18:06:15.957553 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:15.957042 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83085553-108f-4d34-b8de-9d0c6e0b13e5-console-oauth-config\") pod \"console-9b8d5bbf-xnjnp\" (UID: \"83085553-108f-4d34-b8de-9d0c6e0b13e5\") " pod="openshift-console/console-9b8d5bbf-xnjnp" Nov 02 18:06:15.957553 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:15.957191 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83085553-108f-4d34-b8de-9d0c6e0b13e5-oauth-serving-cert\") pod \"console-9b8d5bbf-xnjnp\" (UID: \"83085553-108f-4d34-b8de-9d0c6e0b13e5\") " pod="openshift-console/console-9b8d5bbf-xnjnp" Nov 02 18:06:15.957553 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:15.957225 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b9p2\" (UniqueName: \"kubernetes.io/projected/83085553-108f-4d34-b8de-9d0c6e0b13e5-kube-api-access-7b9p2\") pod \"console-9b8d5bbf-xnjnp\" (UID: \"83085553-108f-4d34-b8de-9d0c6e0b13e5\") " pod="openshift-console/console-9b8d5bbf-xnjnp" Nov 02 18:06:15.957553 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:15.957232 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83085553-108f-4d34-b8de-9d0c6e0b13e5-tmp\") pod \"console-9b8d5bbf-xnjnp\" (UID: \"83085553-108f-4d34-b8de-9d0c6e0b13e5\") " pod="openshift-console/console-9b8d5bbf-xnjnp" Nov 02 18:06:15.958850 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:15.957759 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83085553-108f-4d34-b8de-9d0c6e0b13e5-trusted-ca-bundle\") pod \"console-9b8d5bbf-xnjnp\" (UID: \"83085553-108f-4d34-b8de-9d0c6e0b13e5\") " pod="openshift-console/console-9b8d5bbf-xnjnp" Nov 02 18:06:15.958850 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:15.958054 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/83085553-108f-4d34-b8de-9d0c6e0b13e5-console-config\") pod \"console-9b8d5bbf-xnjnp\" (UID: \"83085553-108f-4d34-b8de-9d0c6e0b13e5\") " pod="openshift-console/console-9b8d5bbf-xnjnp" Nov 02 18:06:15.958850 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:15.958443 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/83085553-108f-4d34-b8de-9d0c6e0b13e5-service-ca\") pod \"console-9b8d5bbf-xnjnp\" (UID: \"83085553-108f-4d34-b8de-9d0c6e0b13e5\") " pod="openshift-console/console-9b8d5bbf-xnjnp" Nov 02 18:06:15.958850 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:15.958627 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83085553-108f-4d34-b8de-9d0c6e0b13e5-oauth-serving-cert\") pod \"console-9b8d5bbf-xnjnp\" (UID: \"83085553-108f-4d34-b8de-9d0c6e0b13e5\") " pod="openshift-console/console-9b8d5bbf-xnjnp" Nov 02 18:06:15.960218 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:15.960183 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83085553-108f-4d34-b8de-9d0c6e0b13e5-console-serving-cert\") pod \"console-9b8d5bbf-xnjnp\" (UID: \"83085553-108f-4d34-b8de-9d0c6e0b13e5\") " pod="openshift-console/console-9b8d5bbf-xnjnp" Nov 02 18:06:15.960398 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:15.960373 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83085553-108f-4d34-b8de-9d0c6e0b13e5-console-oauth-config\") pod \"console-9b8d5bbf-xnjnp\" (UID: \"83085553-108f-4d34-b8de-9d0c6e0b13e5\") " pod="openshift-console/console-9b8d5bbf-xnjnp" Nov 02 18:06:15.983967 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:15.983933 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b9p2\" (UniqueName: \"kubernetes.io/projected/83085553-108f-4d34-b8de-9d0c6e0b13e5-kube-api-access-7b9p2\") pod \"console-9b8d5bbf-xnjnp\" (UID: \"83085553-108f-4d34-b8de-9d0c6e0b13e5\") " pod="openshift-console/console-9b8d5bbf-xnjnp" Nov 02 18:06:16.079283 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:16.079249 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9b8d5bbf-xnjnp" Nov 02 18:06:16.423922 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:06:16.423886 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83085553_108f_4d34_b8de_9d0c6e0b13e5.slice/crio-e39d9b01f015e04fece5789da1105cc03e18cb9dd103de8b65c87aef2faaaf19 WatchSource:0}: Error finding container e39d9b01f015e04fece5789da1105cc03e18cb9dd103de8b65c87aef2faaaf19: Status 404 returned error can't find the container with id e39d9b01f015e04fece5789da1105cc03e18cb9dd103de8b65c87aef2faaaf19 Nov 02 18:06:16.426741 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:16.426715 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9b8d5bbf-xnjnp"] Nov 02 18:06:16.926883 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:16.926843 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9b8d5bbf-xnjnp" event={"ID":"83085553-108f-4d34-b8de-9d0c6e0b13e5","Type":"ContainerStarted","Data":"773191a50fdcf4ae1309a440e1737b7705be93ba616179653ccf5cb035fd7e22"} Nov 02 18:06:16.927298 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:16.926896 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9b8d5bbf-xnjnp" event={"ID":"83085553-108f-4d34-b8de-9d0c6e0b13e5","Type":"ContainerStarted","Data":"e39d9b01f015e04fece5789da1105cc03e18cb9dd103de8b65c87aef2faaaf19"} Nov 02 18:06:16.977115 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:16.977064 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-9b8d5bbf-xnjnp" podStartSLOduration=1.9770495860000001 podStartE2EDuration="1.977049586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:06:16.972097069 +0000 UTC m=+2194.667452562" watchObservedRunningTime="2026-11-02 18:06:16.977049586 +0000 UTC m=+2194.672405058" Nov 02 18:06:19.774750 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:19.774712 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rt5cp_1b5ee10f-87a1-4168-904e-ba233f304fe3/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:26.080279 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:26.080193 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9b8d5bbf-xnjnp" Nov 02 18:06:26.080279 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:26.080252 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9b8d5bbf-xnjnp" Nov 02 18:06:26.085348 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:26.085260 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9b8d5bbf-xnjnp" Nov 02 18:06:27.086245 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:27.086214 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9b8d5bbf-xnjnp" Nov 02 18:06:27.174655 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:27.174597 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8564db559-mnvjc"] Nov 02 18:06:29.936774 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:29.936732 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rt5cp_1b5ee10f-87a1-4168-904e-ba233f304fe3/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:39.185808 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:06:39.185773 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:06:40.098833 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:40.098798 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rt5cp_1b5ee10f-87a1-4168-904e-ba233f304fe3/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:50.262212 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:50.262175 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rt5cp_1b5ee10f-87a1-4168-904e-ba233f304fe3/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:52.201387 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:52.201347 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-8564db559-mnvjc" podUID="f57ff739-d58d-4f6b-b6f8-afed2b4f6afe" containerName="console" containerID="cri-o://44c733729df52fe4036eca347a2eedbfcd380e5a6a2150f22f3b92084cfd237a" gracePeriod=15 Nov 02 18:06:52.491435 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:52.491410 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8564db559-mnvjc" Nov 02 18:06:52.555002 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:52.554962 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-tmp\") pod \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\" (UID: \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\") " Nov 02 18:06:52.555197 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:52.555017 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-kube-api-access-89fl2\" (UniqueName: \"kubernetes.io/projected/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-kube-api-access-89fl2\") pod \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\" (UID: \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\") " Nov 02 18:06:52.555275 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:52.555220 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-console-serving-cert\") pod \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\" (UID: \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\") " Nov 02 18:06:52.555363 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:52.555274 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-tmp" pod "f57ff739-d58d-4f6b-b6f8-afed2b4f6afe" (UID: "f57ff739-d58d-4f6b-b6f8-afed2b4f6afe"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:06:52.555426 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:52.555365 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-trusted-ca-bundle\") pod \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\" (UID: \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\") " Nov 02 18:06:52.555426 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:52.555401 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-service-ca\" (UniqueName: \"kubernetes.io/configmap/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-service-ca\") pod \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\" (UID: \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\") " Nov 02 18:06:52.555545 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:52.555477 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-console-oauth-config\") pod \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\" (UID: \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\") " Nov 02 18:06:52.555603 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:52.555539 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-oauth-serving-cert\") pod \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\" (UID: \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\") " Nov 02 18:06:52.555603 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:52.555596 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-console-config\" (UniqueName: \"kubernetes.io/configmap/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-console-config\") pod \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\" (UID: \"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe\") " Nov 02 18:06:52.555883 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:52.555853 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-trusted-ca-bundle" pod "f57ff739-d58d-4f6b-b6f8-afed2b4f6afe" (UID: "f57ff739-d58d-4f6b-b6f8-afed2b4f6afe"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:06:52.556766 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:52.556739 4471 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-tmp\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:06:52.556899 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:52.556769 4471 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-trusted-ca-bundle\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:06:52.557255 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:52.557231 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-oauth-serving-cert" pod "f57ff739-d58d-4f6b-b6f8-afed2b4f6afe" (UID: "f57ff739-d58d-4f6b-b6f8-afed2b4f6afe"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:06:52.557648 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:52.557620 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-console-config" pod "f57ff739-d58d-4f6b-b6f8-afed2b4f6afe" (UID: "f57ff739-d58d-4f6b-b6f8-afed2b4f6afe"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:06:52.558333 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:52.558093 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-console-oauth-config" pod "f57ff739-d58d-4f6b-b6f8-afed2b4f6afe" (UID: "f57ff739-d58d-4f6b-b6f8-afed2b4f6afe"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:06:52.558465 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:52.558439 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-console-serving-cert" pod "f57ff739-d58d-4f6b-b6f8-afed2b4f6afe" (UID: "f57ff739-d58d-4f6b-b6f8-afed2b4f6afe"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:06:52.558642 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:52.558546 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-kube-api-access-89fl2" pod "f57ff739-d58d-4f6b-b6f8-afed2b4f6afe" (UID: "f57ff739-d58d-4f6b-b6f8-afed2b4f6afe"). InnerVolumeSpecName "kube-api-access-89fl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:06:52.558642 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:52.558574 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-service-ca" pod "f57ff739-d58d-4f6b-b6f8-afed2b4f6afe" (UID: "f57ff739-d58d-4f6b-b6f8-afed2b4f6afe"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:06:52.662499 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:52.662454 4471 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-console-serving-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:06:52.662499 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:52.662491 4471 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:06:52.662499 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:52.662505 4471 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-console-oauth-config\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:06:52.662779 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:52.662520 4471 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-oauth-serving-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:06:52.662779 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:52.662536 4471 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-console-config\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:06:52.662779 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:52.662549 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89fl2\" (UniqueName: \"kubernetes.io/projected/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe-kube-api-access-89fl2\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:06:53.301023 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:53.300983 4471 generic.go:356] "Generic (PLEG): container finished" podID="f57ff739-d58d-4f6b-b6f8-afed2b4f6afe" containerID="44c733729df52fe4036eca347a2eedbfcd380e5a6a2150f22f3b92084cfd237a" exitCode=0 Nov 02 18:06:53.301514 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:53.301035 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8564db559-mnvjc" Nov 02 18:06:53.301514 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:53.301066 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8564db559-mnvjc" event={"ID":"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe","Type":"ContainerDied","Data":"44c733729df52fe4036eca347a2eedbfcd380e5a6a2150f22f3b92084cfd237a"} Nov 02 18:06:53.301514 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:53.301110 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8564db559-mnvjc" event={"ID":"f57ff739-d58d-4f6b-b6f8-afed2b4f6afe","Type":"ContainerDied","Data":"3fed70d86bf78af91d6cc98f7b4bbba82168129585dafb2c56d16d6c02a59299"} Nov 02 18:06:53.301514 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:53.301135 4471 scope.go:122] "RemoveContainer" containerID="44c733729df52fe4036eca347a2eedbfcd380e5a6a2150f22f3b92084cfd237a" Nov 02 18:06:53.313401 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:53.313380 4471 scope.go:122] "RemoveContainer" containerID="44c733729df52fe4036eca347a2eedbfcd380e5a6a2150f22f3b92084cfd237a" Nov 02 18:06:53.313726 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:06:53.313707 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44c733729df52fe4036eca347a2eedbfcd380e5a6a2150f22f3b92084cfd237a\": container with ID starting with 44c733729df52fe4036eca347a2eedbfcd380e5a6a2150f22f3b92084cfd237a not found: ID does not exist" containerID="44c733729df52fe4036eca347a2eedbfcd380e5a6a2150f22f3b92084cfd237a" Nov 02 18:06:53.313799 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:53.313739 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44c733729df52fe4036eca347a2eedbfcd380e5a6a2150f22f3b92084cfd237a"} err="failed to get container status \"44c733729df52fe4036eca347a2eedbfcd380e5a6a2150f22f3b92084cfd237a\": rpc error: code = NotFound desc = could not find container \"44c733729df52fe4036eca347a2eedbfcd380e5a6a2150f22f3b92084cfd237a\": container with ID starting with 44c733729df52fe4036eca347a2eedbfcd380e5a6a2150f22f3b92084cfd237a not found: ID does not exist" Nov 02 18:06:53.358474 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:53.358449 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8564db559-mnvjc"] Nov 02 18:06:53.370719 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:53.370692 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8564db559-mnvjc"] Nov 02 18:06:54.444050 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:06:54.444016 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f57ff739-d58d-4f6b-b6f8-afed2b4f6afe" path="/var/lib/kubelet/pods/f57ff739-d58d-4f6b-b6f8-afed2b4f6afe/volumes" Nov 02 18:07:00.430441 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:00.430406 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rt5cp_1b5ee10f-87a1-4168-904e-ba233f304fe3/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:01.970654 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:01.970618 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5"] Nov 02 18:07:01.971601 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:01.971575 4471 cpu_manager.go:414] "containerMap: removing container" podUID="f57ff739-d58d-4f6b-b6f8-afed2b4f6afe" containerName="console" Nov 02 18:07:01.971601 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:01.971599 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f57ff739-d58d-4f6b-b6f8-afed2b4f6afe" containerName="console" Nov 02 18:07:01.971885 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:01.971869 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="f57ff739-d58d-4f6b-b6f8-afed2b4f6afe" containerName="console" Nov 02 18:07:01.977368 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:01.977314 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5" Nov 02 18:07:02.004188 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:02.004155 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5"] Nov 02 18:07:02.007933 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:02.007909 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-557feb\"/\"kube-root-ca.crt\"" Nov 02 18:07:02.008070 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:02.007957 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-557feb\"/\"openshift-service-ca.crt\"" Nov 02 18:07:02.008070 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:02.008002 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-557feb\"/\"default-dockercfg-4qpw2\"" Nov 02 18:07:02.008070 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:02.008017 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-557feb\"/\"auth-test-model-557feb-kserve-self-signed-certs\"" Nov 02 18:07:02.038208 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:02.038178 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abbd398c-ad2f-4d13-91d0-fbcf97802571-home\") pod \"auth-test-model-557feb-kserve-55948fb75c-cxhd5\" (UID: \"abbd398c-ad2f-4d13-91d0-fbcf97802571\") " pod="ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5" Nov 02 18:07:02.038416 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:02.038327 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abbd398c-ad2f-4d13-91d0-fbcf97802571-tmp-dir\") pod \"auth-test-model-557feb-kserve-55948fb75c-cxhd5\" (UID: \"abbd398c-ad2f-4d13-91d0-fbcf97802571\") " pod="ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5" Nov 02 18:07:02.038501 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:02.038425 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd2tx\" (UniqueName: \"kubernetes.io/projected/abbd398c-ad2f-4d13-91d0-fbcf97802571-kube-api-access-qd2tx\") pod \"auth-test-model-557feb-kserve-55948fb75c-cxhd5\" (UID: \"abbd398c-ad2f-4d13-91d0-fbcf97802571\") " pod="ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5" Nov 02 18:07:02.038565 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:02.038518 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abbd398c-ad2f-4d13-91d0-fbcf97802571-dshm\") pod \"auth-test-model-557feb-kserve-55948fb75c-cxhd5\" (UID: \"abbd398c-ad2f-4d13-91d0-fbcf97802571\") " pod="ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5" Nov 02 18:07:02.038616 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:02.038563 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abbd398c-ad2f-4d13-91d0-fbcf97802571-tls-certs\") pod \"auth-test-model-557feb-kserve-55948fb75c-cxhd5\" (UID: \"abbd398c-ad2f-4d13-91d0-fbcf97802571\") " pod="ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5" Nov 02 18:07:02.038999 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:02.038977 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abbd398c-ad2f-4d13-91d0-fbcf97802571-model-cache\") pod \"auth-test-model-557feb-kserve-55948fb75c-cxhd5\" (UID: \"abbd398c-ad2f-4d13-91d0-fbcf97802571\") " pod="ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5" Nov 02 18:07:02.140688 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:02.140648 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abbd398c-ad2f-4d13-91d0-fbcf97802571-model-cache\") pod \"auth-test-model-557feb-kserve-55948fb75c-cxhd5\" (UID: \"abbd398c-ad2f-4d13-91d0-fbcf97802571\") " pod="ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5" Nov 02 18:07:02.140904 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:02.140700 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abbd398c-ad2f-4d13-91d0-fbcf97802571-home\") pod \"auth-test-model-557feb-kserve-55948fb75c-cxhd5\" (UID: \"abbd398c-ad2f-4d13-91d0-fbcf97802571\") " pod="ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5" Nov 02 18:07:02.140904 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:02.140756 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abbd398c-ad2f-4d13-91d0-fbcf97802571-tmp-dir\") pod \"auth-test-model-557feb-kserve-55948fb75c-cxhd5\" (UID: \"abbd398c-ad2f-4d13-91d0-fbcf97802571\") " pod="ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5" Nov 02 18:07:02.140904 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:02.140837 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd2tx\" (UniqueName: \"kubernetes.io/projected/abbd398c-ad2f-4d13-91d0-fbcf97802571-kube-api-access-qd2tx\") pod \"auth-test-model-557feb-kserve-55948fb75c-cxhd5\" (UID: \"abbd398c-ad2f-4d13-91d0-fbcf97802571\") " pod="ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5" Nov 02 18:07:02.140904 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:02.140885 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abbd398c-ad2f-4d13-91d0-fbcf97802571-dshm\") pod \"auth-test-model-557feb-kserve-55948fb75c-cxhd5\" (UID: \"abbd398c-ad2f-4d13-91d0-fbcf97802571\") " pod="ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5" Nov 02 18:07:02.141152 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:02.140915 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abbd398c-ad2f-4d13-91d0-fbcf97802571-tls-certs\") pod \"auth-test-model-557feb-kserve-55948fb75c-cxhd5\" (UID: \"abbd398c-ad2f-4d13-91d0-fbcf97802571\") " pod="ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5" Nov 02 18:07:02.141152 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:02.141119 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abbd398c-ad2f-4d13-91d0-fbcf97802571-model-cache\") pod \"auth-test-model-557feb-kserve-55948fb75c-cxhd5\" (UID: \"abbd398c-ad2f-4d13-91d0-fbcf97802571\") " pod="ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5" Nov 02 18:07:02.141230 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:02.141161 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abbd398c-ad2f-4d13-91d0-fbcf97802571-home\") pod \"auth-test-model-557feb-kserve-55948fb75c-cxhd5\" (UID: \"abbd398c-ad2f-4d13-91d0-fbcf97802571\") " pod="ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5" Nov 02 18:07:02.141268 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:02.141249 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abbd398c-ad2f-4d13-91d0-fbcf97802571-tmp-dir\") pod \"auth-test-model-557feb-kserve-55948fb75c-cxhd5\" (UID: \"abbd398c-ad2f-4d13-91d0-fbcf97802571\") " pod="ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5" Nov 02 18:07:02.143258 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:02.143233 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abbd398c-ad2f-4d13-91d0-fbcf97802571-dshm\") pod \"auth-test-model-557feb-kserve-55948fb75c-cxhd5\" (UID: \"abbd398c-ad2f-4d13-91d0-fbcf97802571\") " pod="ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5" Nov 02 18:07:02.143544 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:02.143527 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abbd398c-ad2f-4d13-91d0-fbcf97802571-tls-certs\") pod \"auth-test-model-557feb-kserve-55948fb75c-cxhd5\" (UID: \"abbd398c-ad2f-4d13-91d0-fbcf97802571\") " pod="ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5" Nov 02 18:07:02.164568 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:02.164540 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd2tx\" (UniqueName: \"kubernetes.io/projected/abbd398c-ad2f-4d13-91d0-fbcf97802571-kube-api-access-qd2tx\") pod \"auth-test-model-557feb-kserve-55948fb75c-cxhd5\" (UID: \"abbd398c-ad2f-4d13-91d0-fbcf97802571\") " pod="ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5" Nov 02 18:07:02.301329 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:02.300635 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5" Nov 02 18:07:02.734832 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:07:02.734802 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabbd398c_ad2f_4d13_91d0_fbcf97802571.slice/crio-22a4ef907eba6a1343d809b3333958721b1189014dc926e60f0916e28b409a2c WatchSource:0}: Error finding container 22a4ef907eba6a1343d809b3333958721b1189014dc926e60f0916e28b409a2c: Status 404 returned error can't find the container with id 22a4ef907eba6a1343d809b3333958721b1189014dc926e60f0916e28b409a2c Nov 02 18:07:02.737461 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:02.737437 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5"] Nov 02 18:07:03.576388 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:03.576351 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5" event={"ID":"abbd398c-ad2f-4d13-91d0-fbcf97802571","Type":"ContainerStarted","Data":"721d153d11b375908cdfa847071e43d44cb8544303522bc1fca5a8349f46731b"} Nov 02 18:07:03.576388 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:03.576395 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5" event={"ID":"abbd398c-ad2f-4d13-91d0-fbcf97802571","Type":"ContainerStarted","Data":"22a4ef907eba6a1343d809b3333958721b1189014dc926e60f0916e28b409a2c"} Nov 02 18:07:03.620204 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:03.620152 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5" podStartSLOduration=2.620137391 podStartE2EDuration="2.620137391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:07:03.619378954 +0000 UTC m=+2241.314734428" watchObservedRunningTime="2026-11-02 18:07:03.620137391 +0000 UTC m=+2241.315492895" Nov 02 18:07:04.301707 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:04.301669 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5" Nov 02 18:07:09.185717 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:07:09.185678 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:07:10.623446 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:10.623395 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rt5cp_1b5ee10f-87a1-4168-904e-ba233f304fe3/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:12.301994 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:12.301960 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5" Nov 02 18:07:12.315125 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:12.315099 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5" Nov 02 18:07:12.699658 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:12.699580 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5" Nov 02 18:07:20.795992 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:20.795958 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rt5cp_1b5ee10f-87a1-4168-904e-ba233f304fe3/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:30.967665 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:30.967625 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rt5cp_1b5ee10f-87a1-4168-904e-ba233f304fe3/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:39.184483 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:07:39.184441 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:07:41.136681 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:41.136644 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rt5cp_1b5ee10f-87a1-4168-904e-ba233f304fe3/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:52.679383 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:52.679298 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8"] Nov 02 18:07:52.711755 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:52.711721 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8" Nov 02 18:07:52.719537 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:52.719504 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8"] Nov 02 18:07:52.740442 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:52.740412 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-227939\"/\"default-dockercfg-mvbxd\"" Nov 02 18:07:52.740591 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:52.740512 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-227939\"/\"openshift-service-ca.crt\"" Nov 02 18:07:52.740591 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:52.740562 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-227939\"/\"kube-root-ca.crt\"" Nov 02 18:07:52.740591 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:52.740561 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-227939\"/\"auth-test-model-227939-kserve-self-signed-certs\"" Nov 02 18:07:52.811811 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:52.811772 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14d9d576-5cbc-4938-82ec-34191e9675a5-tls-certs\") pod \"auth-test-model-227939-kserve-69d8bb7cc-4fsx8\" (UID: \"14d9d576-5cbc-4938-82ec-34191e9675a5\") " pod="ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8" Nov 02 18:07:52.811989 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:52.811883 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14d9d576-5cbc-4938-82ec-34191e9675a5-home\") pod \"auth-test-model-227939-kserve-69d8bb7cc-4fsx8\" (UID: \"14d9d576-5cbc-4938-82ec-34191e9675a5\") " pod="ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8" Nov 02 18:07:52.812100 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:52.812074 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14d9d576-5cbc-4938-82ec-34191e9675a5-dshm\") pod \"auth-test-model-227939-kserve-69d8bb7cc-4fsx8\" (UID: \"14d9d576-5cbc-4938-82ec-34191e9675a5\") " pod="ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8" Nov 02 18:07:52.812289 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:52.812263 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14d9d576-5cbc-4938-82ec-34191e9675a5-tmp-dir\") pod \"auth-test-model-227939-kserve-69d8bb7cc-4fsx8\" (UID: \"14d9d576-5cbc-4938-82ec-34191e9675a5\") " pod="ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8" Nov 02 18:07:52.812396 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:52.812382 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14d9d576-5cbc-4938-82ec-34191e9675a5-model-cache\") pod \"auth-test-model-227939-kserve-69d8bb7cc-4fsx8\" (UID: \"14d9d576-5cbc-4938-82ec-34191e9675a5\") " pod="ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8" Nov 02 18:07:52.812496 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:52.812481 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gmbt\" (UniqueName: \"kubernetes.io/projected/14d9d576-5cbc-4938-82ec-34191e9675a5-kube-api-access-9gmbt\") pod \"auth-test-model-227939-kserve-69d8bb7cc-4fsx8\" (UID: \"14d9d576-5cbc-4938-82ec-34191e9675a5\") " pod="ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8" Nov 02 18:07:52.926901 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:52.926850 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14d9d576-5cbc-4938-82ec-34191e9675a5-tmp-dir\") pod \"auth-test-model-227939-kserve-69d8bb7cc-4fsx8\" (UID: \"14d9d576-5cbc-4938-82ec-34191e9675a5\") " pod="ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8" Nov 02 18:07:52.927883 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:52.927852 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14d9d576-5cbc-4938-82ec-34191e9675a5-model-cache\") pod \"auth-test-model-227939-kserve-69d8bb7cc-4fsx8\" (UID: \"14d9d576-5cbc-4938-82ec-34191e9675a5\") " pod="ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8" Nov 02 18:07:52.928620 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:52.928596 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gmbt\" (UniqueName: \"kubernetes.io/projected/14d9d576-5cbc-4938-82ec-34191e9675a5-kube-api-access-9gmbt\") pod \"auth-test-model-227939-kserve-69d8bb7cc-4fsx8\" (UID: \"14d9d576-5cbc-4938-82ec-34191e9675a5\") " pod="ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8" Nov 02 18:07:52.928875 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:52.928857 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14d9d576-5cbc-4938-82ec-34191e9675a5-tls-certs\") pod \"auth-test-model-227939-kserve-69d8bb7cc-4fsx8\" (UID: \"14d9d576-5cbc-4938-82ec-34191e9675a5\") " pod="ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8" Nov 02 18:07:52.929009 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:52.928996 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14d9d576-5cbc-4938-82ec-34191e9675a5-home\") pod \"auth-test-model-227939-kserve-69d8bb7cc-4fsx8\" (UID: \"14d9d576-5cbc-4938-82ec-34191e9675a5\") " pod="ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8" Nov 02 18:07:52.949064 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:52.948986 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14d9d576-5cbc-4938-82ec-34191e9675a5-dshm\") pod \"auth-test-model-227939-kserve-69d8bb7cc-4fsx8\" (UID: \"14d9d576-5cbc-4938-82ec-34191e9675a5\") " pod="ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8" Nov 02 18:07:52.949628 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:52.928417 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14d9d576-5cbc-4938-82ec-34191e9675a5-model-cache\") pod \"auth-test-model-227939-kserve-69d8bb7cc-4fsx8\" (UID: \"14d9d576-5cbc-4938-82ec-34191e9675a5\") " pod="ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8" Nov 02 18:07:52.949786 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:52.932580 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14d9d576-5cbc-4938-82ec-34191e9675a5-home\") pod \"auth-test-model-227939-kserve-69d8bb7cc-4fsx8\" (UID: \"14d9d576-5cbc-4938-82ec-34191e9675a5\") " pod="ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8" Nov 02 18:07:52.949883 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:52.940392 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14d9d576-5cbc-4938-82ec-34191e9675a5-tls-certs\") pod \"auth-test-model-227939-kserve-69d8bb7cc-4fsx8\" (UID: \"14d9d576-5cbc-4938-82ec-34191e9675a5\") " pod="ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8" Nov 02 18:07:52.949972 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:52.927733 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14d9d576-5cbc-4938-82ec-34191e9675a5-tmp-dir\") pod \"auth-test-model-227939-kserve-69d8bb7cc-4fsx8\" (UID: \"14d9d576-5cbc-4938-82ec-34191e9675a5\") " pod="ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8" Nov 02 18:07:52.963491 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:52.963461 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14d9d576-5cbc-4938-82ec-34191e9675a5-dshm\") pod \"auth-test-model-227939-kserve-69d8bb7cc-4fsx8\" (UID: \"14d9d576-5cbc-4938-82ec-34191e9675a5\") " pod="ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8" Nov 02 18:07:52.967324 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:52.963938 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gmbt\" (UniqueName: \"kubernetes.io/projected/14d9d576-5cbc-4938-82ec-34191e9675a5-kube-api-access-9gmbt\") pod \"auth-test-model-227939-kserve-69d8bb7cc-4fsx8\" (UID: \"14d9d576-5cbc-4938-82ec-34191e9675a5\") " pod="ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8" Nov 02 18:07:53.032259 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:53.032223 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8" Nov 02 18:07:53.453508 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:07:53.453438 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14d9d576_5cbc_4938_82ec_34191e9675a5.slice/crio-430f944cc9079f9fb50cb014618c9e86e3827a1de5e6abb8c2f9369613b70c59 WatchSource:0}: Error finding container 430f944cc9079f9fb50cb014618c9e86e3827a1de5e6abb8c2f9369613b70c59: Status 404 returned error can't find the container with id 430f944cc9079f9fb50cb014618c9e86e3827a1de5e6abb8c2f9369613b70c59 Nov 02 18:07:53.456832 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:53.456805 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8"] Nov 02 18:07:54.197094 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:54.197051 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8" event={"ID":"14d9d576-5cbc-4938-82ec-34191e9675a5","Type":"ContainerStarted","Data":"e0c124a9e77296660ce1306cd1a02806cc7e72aedb00e2ba550d77e72539b529"} Nov 02 18:07:54.197523 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:54.197106 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8" event={"ID":"14d9d576-5cbc-4938-82ec-34191e9675a5","Type":"ContainerStarted","Data":"430f944cc9079f9fb50cb014618c9e86e3827a1de5e6abb8c2f9369613b70c59"} Nov 02 18:07:54.246446 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:54.246392 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8" podStartSLOduration=2.246377667 podStartE2EDuration="2.246377667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:07:54.245525361 +0000 UTC m=+2291.940880831" watchObservedRunningTime="2026-11-02 18:07:54.246377667 +0000 UTC m=+2291.941733139" Nov 02 18:07:55.033452 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:07:55.033375 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8" Nov 02 18:08:03.033291 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:03.033242 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8" Nov 02 18:08:03.046121 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:03.046095 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8" Nov 02 18:08:03.397686 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:03.397602 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8" Nov 02 18:08:09.185787 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:08:09.185751 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:08:14.530483 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:14.530448 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-st8b5"] Nov 02 18:08:14.535650 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:14.535625 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-st8b5" Nov 02 18:08:14.553565 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:14.553537 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-st8b5"] Nov 02 18:08:14.645691 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:14.645661 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-gclvz\"" Nov 02 18:08:14.671761 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:14.671727 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmk8n\" (UniqueName: \"kubernetes.io/projected/4fd49e19-2141-451a-9c0b-2616cf2d6c71-kube-api-access-hmk8n\") pod \"downloads-6dd789c8b7-st8b5\" (UID: \"4fd49e19-2141-451a-9c0b-2616cf2d6c71\") " pod="openshift-console/downloads-6dd789c8b7-st8b5" Nov 02 18:08:14.671994 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:14.671927 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fd49e19-2141-451a-9c0b-2616cf2d6c71-tmp\") pod \"downloads-6dd789c8b7-st8b5\" (UID: \"4fd49e19-2141-451a-9c0b-2616cf2d6c71\") " pod="openshift-console/downloads-6dd789c8b7-st8b5" Nov 02 18:08:14.774061 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:14.774028 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fd49e19-2141-451a-9c0b-2616cf2d6c71-tmp\") pod \"downloads-6dd789c8b7-st8b5\" (UID: \"4fd49e19-2141-451a-9c0b-2616cf2d6c71\") " pod="openshift-console/downloads-6dd789c8b7-st8b5" Nov 02 18:08:14.774287 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:14.774269 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmk8n\" (UniqueName: \"kubernetes.io/projected/4fd49e19-2141-451a-9c0b-2616cf2d6c71-kube-api-access-hmk8n\") pod \"downloads-6dd789c8b7-st8b5\" (UID: \"4fd49e19-2141-451a-9c0b-2616cf2d6c71\") " pod="openshift-console/downloads-6dd789c8b7-st8b5" Nov 02 18:08:14.774534 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:14.774512 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fd49e19-2141-451a-9c0b-2616cf2d6c71-tmp\") pod \"downloads-6dd789c8b7-st8b5\" (UID: \"4fd49e19-2141-451a-9c0b-2616cf2d6c71\") " pod="openshift-console/downloads-6dd789c8b7-st8b5" Nov 02 18:08:14.798704 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:14.798643 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmk8n\" (UniqueName: \"kubernetes.io/projected/4fd49e19-2141-451a-9c0b-2616cf2d6c71-kube-api-access-hmk8n\") pod \"downloads-6dd789c8b7-st8b5\" (UID: \"4fd49e19-2141-451a-9c0b-2616cf2d6c71\") " pod="openshift-console/downloads-6dd789c8b7-st8b5" Nov 02 18:08:14.853251 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:14.853215 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-st8b5" Nov 02 18:08:15.176637 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:08:15.176597 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd49e19_2141_451a_9c0b_2616cf2d6c71.slice/crio-2371b3f57e6dbe036c2775f7e735f363ea04d4a427008cc4b8d29a12bdccc4c5 WatchSource:0}: Error finding container 2371b3f57e6dbe036c2775f7e735f363ea04d4a427008cc4b8d29a12bdccc4c5: Status 404 returned error can't find the container with id 2371b3f57e6dbe036c2775f7e735f363ea04d4a427008cc4b8d29a12bdccc4c5 Nov 02 18:08:15.179897 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:15.179869 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-st8b5"] Nov 02 18:08:15.534523 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:15.534483 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-st8b5" event={"ID":"4fd49e19-2141-451a-9c0b-2616cf2d6c71","Type":"ContainerStarted","Data":"2371b3f57e6dbe036c2775f7e735f363ea04d4a427008cc4b8d29a12bdccc4c5"} Nov 02 18:08:24.564238 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:24.564199 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1c66dee-ht7cn"] Nov 02 18:08:24.570075 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:24.570050 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c66dee-ht7cn" Nov 02 18:08:24.587646 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:24.587618 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnrqh\" (UniqueName: \"kubernetes.io/projected/e9f3b98a-f972-4127-a97f-d8c1d1937394-kube-api-access-gnrqh\") pod \"maas-api-revoke-sub-b1c66dee-ht7cn\" (UID: \"e9f3b98a-f972-4127-a97f-d8c1d1937394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c66dee-ht7cn" Nov 02 18:08:24.587808 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:24.587743 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1c66dee-ht7cn"] Nov 02 18:08:24.587876 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:24.587836 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9f3b98a-f972-4127-a97f-d8c1d1937394-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1c66dee-ht7cn\" (UID: \"e9f3b98a-f972-4127-a97f-d8c1d1937394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c66dee-ht7cn" Nov 02 18:08:24.587935 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:24.587880 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9f3b98a-f972-4127-a97f-d8c1d1937394-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1c66dee-ht7cn\" (UID: \"e9f3b98a-f972-4127-a97f-d8c1d1937394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c66dee-ht7cn" Nov 02 18:08:24.593813 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:24.593791 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:08:24.593967 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:24.593922 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 18:08:24.689995 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:24.689962 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9f3b98a-f972-4127-a97f-d8c1d1937394-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1c66dee-ht7cn\" (UID: \"e9f3b98a-f972-4127-a97f-d8c1d1937394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c66dee-ht7cn" Nov 02 18:08:24.690185 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:24.690014 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9f3b98a-f972-4127-a97f-d8c1d1937394-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1c66dee-ht7cn\" (UID: \"e9f3b98a-f972-4127-a97f-d8c1d1937394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c66dee-ht7cn" Nov 02 18:08:24.690368 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:24.690346 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnrqh\" (UniqueName: \"kubernetes.io/projected/e9f3b98a-f972-4127-a97f-d8c1d1937394-kube-api-access-gnrqh\") pod \"maas-api-revoke-sub-b1c66dee-ht7cn\" (UID: \"e9f3b98a-f972-4127-a97f-d8c1d1937394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c66dee-ht7cn" Nov 02 18:08:24.690803 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:24.690779 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9f3b98a-f972-4127-a97f-d8c1d1937394-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1c66dee-ht7cn\" (UID: \"e9f3b98a-f972-4127-a97f-d8c1d1937394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c66dee-ht7cn" Nov 02 18:08:24.692765 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:24.692740 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9f3b98a-f972-4127-a97f-d8c1d1937394-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1c66dee-ht7cn\" (UID: \"e9f3b98a-f972-4127-a97f-d8c1d1937394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c66dee-ht7cn" Nov 02 18:08:24.714611 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:24.714587 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnrqh\" (UniqueName: \"kubernetes.io/projected/e9f3b98a-f972-4127-a97f-d8c1d1937394-kube-api-access-gnrqh\") pod \"maas-api-revoke-sub-b1c66dee-ht7cn\" (UID: \"e9f3b98a-f972-4127-a97f-d8c1d1937394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c66dee-ht7cn" Nov 02 18:08:24.892117 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:24.892026 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c66dee-ht7cn" Nov 02 18:08:25.216170 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:08:25.216132 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f3b98a_f972_4127_a97f_d8c1d1937394.slice/crio-7f2c89119e71f5162bdd7278ecca948c882e3146f4ef93687a97176c8df22214 WatchSource:0}: Error finding container 7f2c89119e71f5162bdd7278ecca948c882e3146f4ef93687a97176c8df22214: Status 404 returned error can't find the container with id 7f2c89119e71f5162bdd7278ecca948c882e3146f4ef93687a97176c8df22214 Nov 02 18:08:25.221803 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:25.221780 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1c66dee-ht7cn"] Nov 02 18:08:25.642198 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:25.642155 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c66dee-ht7cn" event={"ID":"e9f3b98a-f972-4127-a97f-d8c1d1937394","Type":"ContainerStarted","Data":"71932218bdacfd0bd3e0827cef38b1fa074e410e5b487855ced20d0b6255042b"} Nov 02 18:08:25.642735 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:25.642232 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c66dee-ht7cn" event={"ID":"e9f3b98a-f972-4127-a97f-d8c1d1937394","Type":"ContainerStarted","Data":"7f2c89119e71f5162bdd7278ecca948c882e3146f4ef93687a97176c8df22214"} Nov 02 18:08:25.687673 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:25.687623 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c66dee-ht7cn" podStartSLOduration=1.687608844 podStartE2EDuration="1.687608844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:08:25.683901574 +0000 UTC m=+2323.379257074" watchObservedRunningTime="2026-11-02 18:08:25.687608844 +0000 UTC m=+2323.382964317" Nov 02 18:08:26.574733 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:26.574694 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rt5cp_1b5ee10f-87a1-4168-904e-ba233f304fe3/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:26.655068 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:26.655028 4471 generic.go:356] "Generic (PLEG): container finished" podID="e9f3b98a-f972-4127-a97f-d8c1d1937394" containerID="71932218bdacfd0bd3e0827cef38b1fa074e410e5b487855ced20d0b6255042b" exitCode=0 Nov 02 18:08:26.655573 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:26.655091 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c66dee-ht7cn" event={"ID":"e9f3b98a-f972-4127-a97f-d8c1d1937394","Type":"ContainerDied","Data":"71932218bdacfd0bd3e0827cef38b1fa074e410e5b487855ced20d0b6255042b"} Nov 02 18:08:27.883542 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:27.883509 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c66dee-ht7cn" Nov 02 18:08:27.991879 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:27.991787 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9f3b98a-f972-4127-a97f-d8c1d1937394-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9f3b98a-f972-4127-a97f-d8c1d1937394-maas-api-xks-ca\") pod \"e9f3b98a-f972-4127-a97f-d8c1d1937394\" (UID: \"e9f3b98a-f972-4127-a97f-d8c1d1937394\") " Nov 02 18:08:27.991879 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:27.991870 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9f3b98a-f972-4127-a97f-d8c1d1937394-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9f3b98a-f972-4127-a97f-d8c1d1937394-maas-api-openshift-ca\") pod \"e9f3b98a-f972-4127-a97f-d8c1d1937394\" (UID: \"e9f3b98a-f972-4127-a97f-d8c1d1937394\") " Nov 02 18:08:27.992119 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:27.992059 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9f3b98a-f972-4127-a97f-d8c1d1937394-kube-api-access-gnrqh\" (UniqueName: \"kubernetes.io/projected/e9f3b98a-f972-4127-a97f-d8c1d1937394-kube-api-access-gnrqh\") pod \"e9f3b98a-f972-4127-a97f-d8c1d1937394\" (UID: \"e9f3b98a-f972-4127-a97f-d8c1d1937394\") " Nov 02 18:08:27.994071 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:27.993982 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9f3b98a-f972-4127-a97f-d8c1d1937394-maas-api-openshift-ca" pod "e9f3b98a-f972-4127-a97f-d8c1d1937394" (UID: "e9f3b98a-f972-4127-a97f-d8c1d1937394"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:08:28.003340 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:28.003273 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f3b98a-f972-4127-a97f-d8c1d1937394-kube-api-access-gnrqh" pod "e9f3b98a-f972-4127-a97f-d8c1d1937394" (UID: "e9f3b98a-f972-4127-a97f-d8c1d1937394"). InnerVolumeSpecName "kube-api-access-gnrqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:08:28.009620 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:28.009587 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9f3b98a-f972-4127-a97f-d8c1d1937394-maas-api-xks-ca" pod "e9f3b98a-f972-4127-a97f-d8c1d1937394" (UID: "e9f3b98a-f972-4127-a97f-d8c1d1937394"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:08:28.095609 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:28.095568 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9f3b98a-f972-4127-a97f-d8c1d1937394-maas-api-xks-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:08:28.095609 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:28.095607 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9f3b98a-f972-4127-a97f-d8c1d1937394-maas-api-openshift-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:08:28.095856 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:28.095623 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnrqh\" (UniqueName: \"kubernetes.io/projected/e9f3b98a-f972-4127-a97f-d8c1d1937394-kube-api-access-gnrqh\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:08:28.679810 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:28.679775 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c66dee-ht7cn" Nov 02 18:08:28.679810 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:28.679789 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c66dee-ht7cn" event={"ID":"e9f3b98a-f972-4127-a97f-d8c1d1937394","Type":"ContainerDied","Data":"7f2c89119e71f5162bdd7278ecca948c882e3146f4ef93687a97176c8df22214"} Nov 02 18:08:28.680028 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:28.679847 4471 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f2c89119e71f5162bdd7278ecca948c882e3146f4ef93687a97176c8df22214" Nov 02 18:08:33.738270 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:33.738175 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-st8b5" event={"ID":"4fd49e19-2141-451a-9c0b-2616cf2d6c71","Type":"ContainerStarted","Data":"c443557208f8c2cada464137155a55c9b42c387fdd2e13538eeea4b3de2a91f7"} Nov 02 18:08:33.739596 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:33.738428 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-st8b5" Nov 02 18:08:33.755338 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:33.755291 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-st8b5" Nov 02 18:08:33.783762 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:33.783700 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-st8b5" podStartSLOduration=1.5437578109999999 podStartE2EDuration="19.783683009s" totalImagesPullingTime="18.239925198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:08:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:08:33.782624051 +0000 UTC m=+2331.477979549" watchObservedRunningTime="2026-11-02 18:08:33.783683009 +0000 UTC m=+2331.479038484" Nov 02 18:08:36.838555 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:36.838511 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rt5cp_1b5ee10f-87a1-4168-904e-ba233f304fe3/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:39.021227 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:39.021177 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8"] Nov 02 18:08:39.499761 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:08:39.499657 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:08:39.577835 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:39.577806 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a161d95e-bmz7c"] Nov 02 18:08:39.579042 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:39.579020 4471 cpu_manager.go:414] "containerMap: removing container" podUID="e9f3b98a-f972-4127-a97f-d8c1d1937394" containerName="revoke-keys" Nov 02 18:08:39.579042 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:39.579042 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9f3b98a-f972-4127-a97f-d8c1d1937394" containerName="revoke-keys" Nov 02 18:08:39.579355 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:39.579294 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9f3b98a-f972-4127-a97f-d8c1d1937394" containerName="revoke-keys" Nov 02 18:08:39.601449 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:39.601417 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a161d95e-bmz7c" Nov 02 18:08:39.607794 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:39.607768 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a161d95e-bmz7c"] Nov 02 18:08:39.621356 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:39.619709 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:08:39.621356 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:39.619892 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 18:08:39.624029 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:39.623994 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5278da20-920d-4ed6-8d05-f477d8d628bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a161d95e-bmz7c\" (UID: \"5278da20-920d-4ed6-8d05-f477d8d628bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a161d95e-bmz7c" Nov 02 18:08:39.624199 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:39.624169 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5278da20-920d-4ed6-8d05-f477d8d628bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a161d95e-bmz7c\" (UID: \"5278da20-920d-4ed6-8d05-f477d8d628bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a161d95e-bmz7c" Nov 02 18:08:39.624349 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:39.624263 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd79m\" (UniqueName: \"kubernetes.io/projected/5278da20-920d-4ed6-8d05-f477d8d628bf-kube-api-access-rd79m\") pod \"maas-api-revoke-sub-a161d95e-bmz7c\" (UID: \"5278da20-920d-4ed6-8d05-f477d8d628bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a161d95e-bmz7c" Nov 02 18:08:39.726799 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:39.726756 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5278da20-920d-4ed6-8d05-f477d8d628bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a161d95e-bmz7c\" (UID: \"5278da20-920d-4ed6-8d05-f477d8d628bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a161d95e-bmz7c" Nov 02 18:08:39.726987 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:39.726959 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5278da20-920d-4ed6-8d05-f477d8d628bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a161d95e-bmz7c\" (UID: \"5278da20-920d-4ed6-8d05-f477d8d628bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a161d95e-bmz7c" Nov 02 18:08:39.727101 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:39.727077 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd79m\" (UniqueName: \"kubernetes.io/projected/5278da20-920d-4ed6-8d05-f477d8d628bf-kube-api-access-rd79m\") pod \"maas-api-revoke-sub-a161d95e-bmz7c\" (UID: \"5278da20-920d-4ed6-8d05-f477d8d628bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a161d95e-bmz7c" Nov 02 18:08:39.727652 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:39.727626 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5278da20-920d-4ed6-8d05-f477d8d628bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a161d95e-bmz7c\" (UID: \"5278da20-920d-4ed6-8d05-f477d8d628bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a161d95e-bmz7c" Nov 02 18:08:39.729766 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:39.729745 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5278da20-920d-4ed6-8d05-f477d8d628bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a161d95e-bmz7c\" (UID: \"5278da20-920d-4ed6-8d05-f477d8d628bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a161d95e-bmz7c" Nov 02 18:08:39.751487 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:39.751431 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd79m\" (UniqueName: \"kubernetes.io/projected/5278da20-920d-4ed6-8d05-f477d8d628bf-kube-api-access-rd79m\") pod \"maas-api-revoke-sub-a161d95e-bmz7c\" (UID: \"5278da20-920d-4ed6-8d05-f477d8d628bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a161d95e-bmz7c" Nov 02 18:08:39.954344 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:39.954266 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a161d95e-bmz7c" Nov 02 18:08:40.311920 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:08:40.311827 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5278da20_920d_4ed6_8d05_f477d8d628bf.slice/crio-d1cc3e7b1637f291963646f565efe251f7d00441934de77986854a658a812350 WatchSource:0}: Error finding container d1cc3e7b1637f291963646f565efe251f7d00441934de77986854a658a812350: Status 404 returned error can't find the container with id d1cc3e7b1637f291963646f565efe251f7d00441934de77986854a658a812350 Nov 02 18:08:40.319408 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:40.317684 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a161d95e-bmz7c"] Nov 02 18:08:40.819630 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:40.819588 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a161d95e-bmz7c" event={"ID":"5278da20-920d-4ed6-8d05-f477d8d628bf","Type":"ContainerStarted","Data":"d1cc3e7b1637f291963646f565efe251f7d00441934de77986854a658a812350"} Nov 02 18:08:41.837607 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:41.837570 4471 generic.go:356] "Generic (PLEG): container finished" podID="5278da20-920d-4ed6-8d05-f477d8d628bf" containerID="6fcf37e8225599702a9d076fe478baaabea9a6ca56abbc5ff21642096c20a9b3" exitCode=0 Nov 02 18:08:41.838108 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:41.837659 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a161d95e-bmz7c" event={"ID":"5278da20-920d-4ed6-8d05-f477d8d628bf","Type":"ContainerDied","Data":"6fcf37e8225599702a9d076fe478baaabea9a6ca56abbc5ff21642096c20a9b3"} Nov 02 18:08:43.074328 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:43.067589 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a161d95e-bmz7c" Nov 02 18:08:43.159245 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:43.159144 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5278da20-920d-4ed6-8d05-f477d8d628bf-kube-api-access-rd79m\" (UniqueName: \"kubernetes.io/projected/5278da20-920d-4ed6-8d05-f477d8d628bf-kube-api-access-rd79m\") pod \"5278da20-920d-4ed6-8d05-f477d8d628bf\" (UID: \"5278da20-920d-4ed6-8d05-f477d8d628bf\") " Nov 02 18:08:43.159444 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:43.159350 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5278da20-920d-4ed6-8d05-f477d8d628bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5278da20-920d-4ed6-8d05-f477d8d628bf-maas-api-openshift-ca\") pod \"5278da20-920d-4ed6-8d05-f477d8d628bf\" (UID: \"5278da20-920d-4ed6-8d05-f477d8d628bf\") " Nov 02 18:08:43.161625 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:43.161597 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5278da20-920d-4ed6-8d05-f477d8d628bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5278da20-920d-4ed6-8d05-f477d8d628bf-maas-api-xks-ca\") pod \"5278da20-920d-4ed6-8d05-f477d8d628bf\" (UID: \"5278da20-920d-4ed6-8d05-f477d8d628bf\") " Nov 02 18:08:43.162559 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:43.162513 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5278da20-920d-4ed6-8d05-f477d8d628bf-kube-api-access-rd79m" pod "5278da20-920d-4ed6-8d05-f477d8d628bf" (UID: "5278da20-920d-4ed6-8d05-f477d8d628bf"). InnerVolumeSpecName "kube-api-access-rd79m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:08:43.162889 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:43.162862 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5278da20-920d-4ed6-8d05-f477d8d628bf-maas-api-openshift-ca" pod "5278da20-920d-4ed6-8d05-f477d8d628bf" (UID: "5278da20-920d-4ed6-8d05-f477d8d628bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:08:43.172241 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:43.172215 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5278da20-920d-4ed6-8d05-f477d8d628bf-maas-api-xks-ca" pod "5278da20-920d-4ed6-8d05-f477d8d628bf" (UID: "5278da20-920d-4ed6-8d05-f477d8d628bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:08:43.264260 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:43.264227 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5278da20-920d-4ed6-8d05-f477d8d628bf-maas-api-xks-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:08:43.264260 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:43.264261 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd79m\" (UniqueName: \"kubernetes.io/projected/5278da20-920d-4ed6-8d05-f477d8d628bf-kube-api-access-rd79m\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:08:43.264476 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:43.264283 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5278da20-920d-4ed6-8d05-f477d8d628bf-maas-api-openshift-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:08:43.857044 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:43.857011 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a161d95e-bmz7c" Nov 02 18:08:43.857220 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:43.857043 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a161d95e-bmz7c" event={"ID":"5278da20-920d-4ed6-8d05-f477d8d628bf","Type":"ContainerDied","Data":"d1cc3e7b1637f291963646f565efe251f7d00441934de77986854a658a812350"} Nov 02 18:08:43.857220 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:43.857094 4471 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1cc3e7b1637f291963646f565efe251f7d00441934de77986854a658a812350" Nov 02 18:08:47.033597 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:47.032094 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rt5cp_1b5ee10f-87a1-4168-904e-ba233f304fe3/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:50.600838 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:50.600806 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bb6d2e-5b8dd8bc9-pfhk7"] Nov 02 18:08:54.086623 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:54.086578 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" podUID="33601c55-3e8f-41d4-a13e-1099a26334f4" containerName="main" containerID="cri-o://b3bbb91b545645cd58ce56e89a345a6ccf85034c9e14cb1a55b31dcc2b7f21dc" gracePeriod=45 Nov 02 18:08:54.413733 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:54.413706 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" Nov 02 18:08:54.493921 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:54.493880 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33601c55-3e8f-41d4-a13e-1099a26334f4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33601c55-3e8f-41d4-a13e-1099a26334f4-model-cache\") pod \"33601c55-3e8f-41d4-a13e-1099a26334f4\" (UID: \"33601c55-3e8f-41d4-a13e-1099a26334f4\") " Nov 02 18:08:54.494116 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:54.494030 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33601c55-3e8f-41d4-a13e-1099a26334f4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33601c55-3e8f-41d4-a13e-1099a26334f4-tmp-dir\") pod \"33601c55-3e8f-41d4-a13e-1099a26334f4\" (UID: \"33601c55-3e8f-41d4-a13e-1099a26334f4\") " Nov 02 18:08:54.494183 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:54.494156 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33601c55-3e8f-41d4-a13e-1099a26334f4-model-cache" pod "33601c55-3e8f-41d4-a13e-1099a26334f4" (UID: "33601c55-3e8f-41d4-a13e-1099a26334f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:08:54.494247 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:54.494215 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33601c55-3e8f-41d4-a13e-1099a26334f4-tls-certs\" (UniqueName: \"kubernetes.io/secret/33601c55-3e8f-41d4-a13e-1099a26334f4-tls-certs\") pod \"33601c55-3e8f-41d4-a13e-1099a26334f4\" (UID: \"33601c55-3e8f-41d4-a13e-1099a26334f4\") " Nov 02 18:08:54.494331 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:54.494267 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33601c55-3e8f-41d4-a13e-1099a26334f4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/33601c55-3e8f-41d4-a13e-1099a26334f4-dshm\") pod \"33601c55-3e8f-41d4-a13e-1099a26334f4\" (UID: \"33601c55-3e8f-41d4-a13e-1099a26334f4\") " Nov 02 18:08:54.494331 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:54.494285 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33601c55-3e8f-41d4-a13e-1099a26334f4-tmp-dir" pod "33601c55-3e8f-41d4-a13e-1099a26334f4" (UID: "33601c55-3e8f-41d4-a13e-1099a26334f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:08:54.494519 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:54.494454 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33601c55-3e8f-41d4-a13e-1099a26334f4-kube-api-access-smndx\" (UniqueName: \"kubernetes.io/projected/33601c55-3e8f-41d4-a13e-1099a26334f4-kube-api-access-smndx\") pod \"33601c55-3e8f-41d4-a13e-1099a26334f4\" (UID: \"33601c55-3e8f-41d4-a13e-1099a26334f4\") " Nov 02 18:08:54.494519 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:54.494492 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33601c55-3e8f-41d4-a13e-1099a26334f4-home\" (UniqueName: \"kubernetes.io/empty-dir/33601c55-3e8f-41d4-a13e-1099a26334f4-home\") pod \"33601c55-3e8f-41d4-a13e-1099a26334f4\" (UID: \"33601c55-3e8f-41d4-a13e-1099a26334f4\") " Nov 02 18:08:54.494784 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:54.494761 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33601c55-3e8f-41d4-a13e-1099a26334f4-home" pod "33601c55-3e8f-41d4-a13e-1099a26334f4" (UID: "33601c55-3e8f-41d4-a13e-1099a26334f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:08:54.495477 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:54.495451 4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33601c55-3e8f-41d4-a13e-1099a26334f4-home\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:08:54.495573 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:54.495484 4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33601c55-3e8f-41d4-a13e-1099a26334f4-model-cache\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:08:54.495573 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:54.495522 4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33601c55-3e8f-41d4-a13e-1099a26334f4-tmp-dir\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:08:54.496696 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:54.496673 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33601c55-3e8f-41d4-a13e-1099a26334f4-tls-certs" pod "33601c55-3e8f-41d4-a13e-1099a26334f4" (UID: "33601c55-3e8f-41d4-a13e-1099a26334f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:08:54.496696 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:54.496686 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33601c55-3e8f-41d4-a13e-1099a26334f4-dshm" pod "33601c55-3e8f-41d4-a13e-1099a26334f4" (UID: "33601c55-3e8f-41d4-a13e-1099a26334f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:08:54.496947 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:54.496923 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33601c55-3e8f-41d4-a13e-1099a26334f4-kube-api-access-smndx" pod "33601c55-3e8f-41d4-a13e-1099a26334f4" (UID: "33601c55-3e8f-41d4-a13e-1099a26334f4"). InnerVolumeSpecName "kube-api-access-smndx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:08:54.597285 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:54.597247 4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33601c55-3e8f-41d4-a13e-1099a26334f4-tls-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:08:54.597285 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:54.597283 4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33601c55-3e8f-41d4-a13e-1099a26334f4-dshm\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:08:54.597285 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:54.597293 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smndx\" (UniqueName: \"kubernetes.io/projected/33601c55-3e8f-41d4-a13e-1099a26334f4-kube-api-access-smndx\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:08:55.192228 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:55.192189 4471 generic.go:356] "Generic (PLEG): container finished" podID="33601c55-3e8f-41d4-a13e-1099a26334f4" containerID="b3bbb91b545645cd58ce56e89a345a6ccf85034c9e14cb1a55b31dcc2b7f21dc" exitCode=0 Nov 02 18:08:55.192704 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:55.192234 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" Nov 02 18:08:55.192704 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:55.192281 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" event={"ID":"33601c55-3e8f-41d4-a13e-1099a26334f4","Type":"ContainerDied","Data":"b3bbb91b545645cd58ce56e89a345a6ccf85034c9e14cb1a55b31dcc2b7f21dc"} Nov 02 18:08:55.192704 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:55.192363 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8" event={"ID":"33601c55-3e8f-41d4-a13e-1099a26334f4","Type":"ContainerDied","Data":"41a9757ced0af9ff2af61f779dce9ef06abfe2e7baa631b46c51933b9aee3941"} Nov 02 18:08:55.192704 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:55.192384 4471 scope.go:122] "RemoveContainer" containerID="b3bbb91b545645cd58ce56e89a345a6ccf85034c9e14cb1a55b31dcc2b7f21dc" Nov 02 18:08:55.204544 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:55.204520 4471 scope.go:122] "RemoveContainer" containerID="b3bbb91b545645cd58ce56e89a345a6ccf85034c9e14cb1a55b31dcc2b7f21dc" Nov 02 18:08:55.204840 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:08:55.204821 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3bbb91b545645cd58ce56e89a345a6ccf85034c9e14cb1a55b31dcc2b7f21dc\": container with ID starting with b3bbb91b545645cd58ce56e89a345a6ccf85034c9e14cb1a55b31dcc2b7f21dc not found: ID does not exist" containerID="b3bbb91b545645cd58ce56e89a345a6ccf85034c9e14cb1a55b31dcc2b7f21dc" Nov 02 18:08:55.204899 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:55.204849 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3bbb91b545645cd58ce56e89a345a6ccf85034c9e14cb1a55b31dcc2b7f21dc"} err="failed to get container status \"b3bbb91b545645cd58ce56e89a345a6ccf85034c9e14cb1a55b31dcc2b7f21dc\": rpc error: code = NotFound desc = could not find container \"b3bbb91b545645cd58ce56e89a345a6ccf85034c9e14cb1a55b31dcc2b7f21dc\": container with ID starting with b3bbb91b545645cd58ce56e89a345a6ccf85034c9e14cb1a55b31dcc2b7f21dc not found: ID does not exist" Nov 02 18:08:55.250239 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:55.250205 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8"] Nov 02 18:08:55.261314 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:55.261276 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-2f6787/test-model-2f6787-kserve-c77466787-vn7z8"] Nov 02 18:08:55.601397 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:55.601357 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb6d2e-5b8dd8bc9-pfhk7" podUID="6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8" containerName="payload-pre-processing" containerID="cri-o://c5bdf79f9882a21a80c93b7bea365f5c57b61be10def4c341913546c12a74db3" gracePeriod=25 Nov 02 18:08:55.916077 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:55.916051 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb6d2e-5b8dd8bc9-pfhk7" Nov 02 18:08:56.031247 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:56.031213 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8-plugins-config-volume\") pod \"6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8\" (UID: \"6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8\") " Nov 02 18:08:56.031566 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:56.031547 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8-kube-api-access-zftlr\" (UniqueName: \"kubernetes.io/projected/6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8-kube-api-access-zftlr\") pod \"6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8\" (UID: \"6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8\") " Nov 02 18:08:56.037718 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:56.037683 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8-plugins-config-volume" pod "6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8" (UID: "6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:08:56.038714 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:56.038690 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8-kube-api-access-zftlr" pod "6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8" (UID: "6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8"). InnerVolumeSpecName "kube-api-access-zftlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:08:56.134159 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:56.134085 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:08:56.134159 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:56.134113 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zftlr\" (UniqueName: \"kubernetes.io/projected/6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8-kube-api-access-zftlr\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:08:56.204465 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:56.204424 4471 generic.go:356] "Generic (PLEG): container finished" podID="6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8" containerID="c5bdf79f9882a21a80c93b7bea365f5c57b61be10def4c341913546c12a74db3" exitCode=0 Nov 02 18:08:56.204943 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:56.204477 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb6d2e-5b8dd8bc9-pfhk7" Nov 02 18:08:56.204943 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:56.204514 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb6d2e-5b8dd8bc9-pfhk7" event={"ID":"6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8","Type":"ContainerDied","Data":"c5bdf79f9882a21a80c93b7bea365f5c57b61be10def4c341913546c12a74db3"} Nov 02 18:08:56.204943 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:56.204584 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb6d2e-5b8dd8bc9-pfhk7" event={"ID":"6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8","Type":"ContainerDied","Data":"6568a670c694f6f7ebb4f927ed9b2cdfd9c9e1afc868264ad5c47869ab8cb152"} Nov 02 18:08:56.204943 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:56.204615 4471 scope.go:122] "RemoveContainer" containerID="c5bdf79f9882a21a80c93b7bea365f5c57b61be10def4c341913546c12a74db3" Nov 02 18:08:56.223415 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:56.223387 4471 scope.go:122] "RemoveContainer" containerID="c5bdf79f9882a21a80c93b7bea365f5c57b61be10def4c341913546c12a74db3" Nov 02 18:08:56.223770 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:08:56.223746 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5bdf79f9882a21a80c93b7bea365f5c57b61be10def4c341913546c12a74db3\": container with ID starting with c5bdf79f9882a21a80c93b7bea365f5c57b61be10def4c341913546c12a74db3 not found: ID does not exist" containerID="c5bdf79f9882a21a80c93b7bea365f5c57b61be10def4c341913546c12a74db3" Nov 02 18:08:56.223836 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:56.223783 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5bdf79f9882a21a80c93b7bea365f5c57b61be10def4c341913546c12a74db3"} err="failed to get container status \"c5bdf79f9882a21a80c93b7bea365f5c57b61be10def4c341913546c12a74db3\": rpc error: code = NotFound desc = could not find container \"c5bdf79f9882a21a80c93b7bea365f5c57b61be10def4c341913546c12a74db3\": container with ID starting with c5bdf79f9882a21a80c93b7bea365f5c57b61be10def4c341913546c12a74db3 not found: ID does not exist" Nov 02 18:08:56.266569 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:56.266539 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bb6d2e-5b8dd8bc9-pfhk7"] Nov 02 18:08:56.279133 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:56.279099 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bb6d2e-5b8dd8bc9-pfhk7"] Nov 02 18:08:56.443808 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:56.443730 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33601c55-3e8f-41d4-a13e-1099a26334f4" path="/var/lib/kubelet/pods/33601c55-3e8f-41d4-a13e-1099a26334f4/volumes" Nov 02 18:08:56.444346 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:08:56.444325 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8" path="/var/lib/kubelet/pods/6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8/volumes" Nov 02 18:09:05.963481 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:05.963443 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn"] Nov 02 18:09:06.350727 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:06.350694 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6787-gfmbn"] Nov 02 18:09:06.351752 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:06.351726 4471 cpu_manager.go:414] "containerMap: removing container" podUID="6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8" containerName="payload-pre-processing" Nov 02 18:09:06.351752 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:06.351750 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8" containerName="payload-pre-processing" Nov 02 18:09:06.351954 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:06.351801 4471 cpu_manager.go:414] "containerMap: removing container" podUID="33601c55-3e8f-41d4-a13e-1099a26334f4" containerName="main" Nov 02 18:09:06.351954 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:06.351809 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33601c55-3e8f-41d4-a13e-1099a26334f4" containerName="main" Nov 02 18:09:06.351954 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:06.351826 4471 cpu_manager.go:414] "containerMap: removing container" podUID="5278da20-920d-4ed6-8d05-f477d8d628bf" containerName="revoke-keys" Nov 02 18:09:06.351954 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:06.351833 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5278da20-920d-4ed6-8d05-f477d8d628bf" containerName="revoke-keys" Nov 02 18:09:06.352152 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:06.352064 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c3ff900-7ee5-40ce-8ba8-ca08e3f666f8" containerName="payload-pre-processing" Nov 02 18:09:06.352152 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:06.352087 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="33601c55-3e8f-41d4-a13e-1099a26334f4" containerName="main" Nov 02 18:09:06.352152 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:06.352098 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="5278da20-920d-4ed6-8d05-f477d8d628bf" containerName="revoke-keys" Nov 02 18:09:06.363509 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:06.363486 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6787-gfmbn" Nov 02 18:09:06.378397 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:06.378360 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6787-gfmbn"] Nov 02 18:09:06.379415 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:06.379392 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:09:06.382928 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:06.382910 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 18:09:06.499951 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:06.499918 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2f6787-gfmbn\" (UID: \"4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6787-gfmbn" Nov 02 18:09:06.500106 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:06.500075 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2f6787-gfmbn\" (UID: \"4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6787-gfmbn" Nov 02 18:09:06.500244 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:06.500233 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmtfl\" (UniqueName: \"kubernetes.io/projected/4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6-kube-api-access-kmtfl\") pod \"maas-api-revoke-keys-e2e-inf-a-2f6787-gfmbn\" (UID: \"4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6787-gfmbn" Nov 02 18:09:06.602875 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:06.602765 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmtfl\" (UniqueName: \"kubernetes.io/projected/4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6-kube-api-access-kmtfl\") pod \"maas-api-revoke-keys-e2e-inf-a-2f6787-gfmbn\" (UID: \"4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6787-gfmbn" Nov 02 18:09:06.603073 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:06.603004 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2f6787-gfmbn\" (UID: \"4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6787-gfmbn" Nov 02 18:09:06.603837 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:06.603812 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2f6787-gfmbn\" (UID: \"4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6787-gfmbn" Nov 02 18:09:06.604823 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:06.604801 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2f6787-gfmbn\" (UID: \"4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6787-gfmbn" Nov 02 18:09:06.607476 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:06.607454 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2f6787-gfmbn\" (UID: \"4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6787-gfmbn" Nov 02 18:09:06.629097 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:06.629071 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmtfl\" (UniqueName: \"kubernetes.io/projected/4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6-kube-api-access-kmtfl\") pod \"maas-api-revoke-keys-e2e-inf-a-2f6787-gfmbn\" (UID: \"4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6787-gfmbn" Nov 02 18:09:06.686395 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:06.686357 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6787-gfmbn" Nov 02 18:09:07.015668 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:09:07.015636 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4172ebfa_094b_41bb_8e6a_9fde4ae2a7a6.slice/crio-cb784a8422bab906b043cf68fb8ac69974b33f624dfd45d2d363ce61051541dc WatchSource:0}: Error finding container cb784a8422bab906b043cf68fb8ac69974b33f624dfd45d2d363ce61051541dc: Status 404 returned error can't find the container with id cb784a8422bab906b043cf68fb8ac69974b33f624dfd45d2d363ce61051541dc Nov 02 18:09:07.019480 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:07.019093 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6787-gfmbn"] Nov 02 18:09:07.335655 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:07.335622 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6787-gfmbn" event={"ID":"4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6","Type":"ContainerStarted","Data":"cb784a8422bab906b043cf68fb8ac69974b33f624dfd45d2d363ce61051541dc"} Nov 02 18:09:08.348412 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:08.348319 4471 generic.go:356] "Generic (PLEG): container finished" podID="4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6" containerID="0edbd470905a8279d7c16310d64cd1451ad049249bcf3994b2baaaf793e426f2" exitCode=0 Nov 02 18:09:08.348412 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:08.348363 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6787-gfmbn" event={"ID":"4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6","Type":"ContainerDied","Data":"0edbd470905a8279d7c16310d64cd1451ad049249bcf3994b2baaaf793e426f2"} Nov 02 18:09:09.185489 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:09:09.185452 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:09:09.550941 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:09.550912 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6787-gfmbn" Nov 02 18:09:09.574956 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:09.574929 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6-maas-api-xks-ca\") pod \"4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6\" (UID: \"4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6\") " Nov 02 18:09:09.575095 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:09.575053 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6-kube-api-access-kmtfl\" (UniqueName: \"kubernetes.io/projected/4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6-kube-api-access-kmtfl\") pod \"4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6\" (UID: \"4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6\") " Nov 02 18:09:09.575169 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:09.575153 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6-maas-api-openshift-ca\") pod \"4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6\" (UID: \"4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6\") " Nov 02 18:09:09.575573 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:09.575544 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6-maas-api-openshift-ca" pod "4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6" (UID: "4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:09:09.575976 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:09.575958 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6-maas-api-openshift-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:09:09.577069 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:09.577046 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6-maas-api-xks-ca" pod "4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6" (UID: "4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:09:09.577233 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:09.577128 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6-kube-api-access-kmtfl" pod "4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6" (UID: "4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6"). InnerVolumeSpecName "kube-api-access-kmtfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:09:09.687327 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:09.682489 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6-maas-api-xks-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:09:09.687327 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:09.682531 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmtfl\" (UniqueName: \"kubernetes.io/projected/4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6-kube-api-access-kmtfl\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:09:10.366619 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:10.366586 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6787-gfmbn" Nov 02 18:09:10.366789 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:10.366626 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6787-gfmbn" event={"ID":"4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6","Type":"ContainerDied","Data":"cb784a8422bab906b043cf68fb8ac69974b33f624dfd45d2d363ce61051541dc"} Nov 02 18:09:10.366789 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:10.366697 4471 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb784a8422bab906b043cf68fb8ac69974b33f624dfd45d2d363ce61051541dc" Nov 02 18:09:16.871736 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:16.871693 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2f6787-6bb9757b5-tmdf8"] Nov 02 18:09:16.924476 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:16.924442 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2f6787-77f84bf7c9-bwsjs"] Nov 02 18:09:20.991962 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:20.991882 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" podUID="b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2" containerName="main" containerID="cri-o://df8f007263a19f356a22aae99a7e61eaa8ff775b92a167473a2e57f1add8ef65" gracePeriod=45 Nov 02 18:09:21.023222 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:21.023182 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" podUID="b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.153:8000/ready\": dial tcp 10.133.0.153:8000: connect: connection refused" Nov 02 18:09:21.302504 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:21.302479 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" Nov 02 18:09:21.385990 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:21.385954 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-home\" (UniqueName: \"kubernetes.io/empty-dir/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-home\") pod \"b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2\" (UID: \"b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2\") " Nov 02 18:09:21.386169 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:21.386064 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-tls-certs\" (UniqueName: \"kubernetes.io/secret/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-tls-certs\") pod \"b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2\" (UID: \"b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2\") " Nov 02 18:09:21.386218 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:21.386185 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-kube-api-access-pp7m6\" (UniqueName: \"kubernetes.io/projected/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-kube-api-access-pp7m6\") pod \"b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2\" (UID: \"b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2\") " Nov 02 18:09:21.386299 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:21.386280 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-tmp-dir\") pod \"b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2\" (UID: \"b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2\") " Nov 02 18:09:21.386477 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:21.386457 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-dshm\") pod \"b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2\" (UID: \"b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2\") " Nov 02 18:09:21.386538 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:21.386280 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-home" pod "b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2" (UID: "b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:09:21.386601 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:21.386567 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-tmp-dir" pod "b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2" (UID: "b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:09:21.386725 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:21.386706 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-model-cache\") pod \"b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2\" (UID: \"b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2\") " Nov 02 18:09:21.387072 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:21.387045 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-model-cache" pod "b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2" (UID: "b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:09:21.387575 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:21.387545 4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-tmp-dir\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:09:21.387575 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:21.387566 4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-model-cache\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:09:21.387753 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:21.387577 4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-home\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:09:21.389240 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:21.389214 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-tls-certs" pod "b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2" (UID: "b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:09:21.389345 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:21.389276 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-kube-api-access-pp7m6" pod "b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2" (UID: "b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2"). InnerVolumeSpecName "kube-api-access-pp7m6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:09:21.389411 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:21.389297 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-dshm" pod "b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2" (UID: "b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:09:21.489709 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:21.489677 4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-dshm\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:09:21.489709 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:21.489709 4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-tls-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:09:21.489905 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:21.489724 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp7m6\" (UniqueName: \"kubernetes.io/projected/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2-kube-api-access-pp7m6\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:09:21.516767 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:21.516676 4471 generic.go:356] "Generic (PLEG): container finished" podID="b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2" containerID="df8f007263a19f356a22aae99a7e61eaa8ff775b92a167473a2e57f1add8ef65" exitCode=0 Nov 02 18:09:21.516767 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:21.516728 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" Nov 02 18:09:21.516941 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:21.516760 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" event={"ID":"b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2","Type":"ContainerDied","Data":"df8f007263a19f356a22aae99a7e61eaa8ff775b92a167473a2e57f1add8ef65"} Nov 02 18:09:21.516941 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:21.516807 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn" event={"ID":"b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2","Type":"ContainerDied","Data":"154437eeab170def1fbf3a73d6df720f12164b5938639ad4430b2a78b8a95aa6"} Nov 02 18:09:21.516941 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:21.516827 4471 scope.go:122] "RemoveContainer" containerID="df8f007263a19f356a22aae99a7e61eaa8ff775b92a167473a2e57f1add8ef65" Nov 02 18:09:21.529396 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:21.529377 4471 scope.go:122] "RemoveContainer" containerID="df8f007263a19f356a22aae99a7e61eaa8ff775b92a167473a2e57f1add8ef65" Nov 02 18:09:21.529703 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:09:21.529683 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df8f007263a19f356a22aae99a7e61eaa8ff775b92a167473a2e57f1add8ef65\": container with ID starting with df8f007263a19f356a22aae99a7e61eaa8ff775b92a167473a2e57f1add8ef65 not found: ID does not exist" containerID="df8f007263a19f356a22aae99a7e61eaa8ff775b92a167473a2e57f1add8ef65" Nov 02 18:09:21.529769 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:21.529712 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df8f007263a19f356a22aae99a7e61eaa8ff775b92a167473a2e57f1add8ef65"} err="failed to get container status \"df8f007263a19f356a22aae99a7e61eaa8ff775b92a167473a2e57f1add8ef65\": rpc error: code = NotFound desc = could not find container \"df8f007263a19f356a22aae99a7e61eaa8ff775b92a167473a2e57f1add8ef65\": container with ID starting with df8f007263a19f356a22aae99a7e61eaa8ff775b92a167473a2e57f1add8ef65 not found: ID does not exist" Nov 02 18:09:21.576669 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:21.576637 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn"] Nov 02 18:09:21.581623 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:21.581600 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-8a209f/test-model-8a209f-kserve-6544f7897f-r69cn"] Nov 02 18:09:21.872248 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:21.872164 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-2f6787-6bb9757b5-tmdf8" podUID="8cae7c13-8d21-49f1-b1d4-ea369ec4e54c" containerName="payload-processing" containerID="cri-o://7f61effc758f800483895725f323ac9c517c2158f2ac33abd5bd3cc3fa9a4bea" gracePeriod=25 Nov 02 18:09:21.925070 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:21.925026 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6787-77f84bf7c9-bwsjs" podUID="721435f4-cdae-4071-9767-5dbf79c5dd3f" containerName="payload-pre-processing" containerID="cri-o://623eafbab03185c9cb82308d3f856ff4f026ffd7af534c76eed53d3f5a130782" gracePeriod=25 Nov 02 18:09:22.131041 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:22.131004 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6787-77f84bf7c9-bwsjs" podUID="721435f4-cdae-4071-9767-5dbf79c5dd3f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:09:22.169477 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:22.169454 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2f6787-6bb9757b5-tmdf8" Nov 02 18:09:22.206668 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:22.206640 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cae7c13-8d21-49f1-b1d4-ea369ec4e54c-kube-api-access-ztvbn\" (UniqueName: \"kubernetes.io/projected/8cae7c13-8d21-49f1-b1d4-ea369ec4e54c-kube-api-access-ztvbn\") pod \"8cae7c13-8d21-49f1-b1d4-ea369ec4e54c\" (UID: \"8cae7c13-8d21-49f1-b1d4-ea369ec4e54c\") " Nov 02 18:09:22.206815 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:22.206801 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cae7c13-8d21-49f1-b1d4-ea369ec4e54c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cae7c13-8d21-49f1-b1d4-ea369ec4e54c-plugins-config-volume\") pod \"8cae7c13-8d21-49f1-b1d4-ea369ec4e54c\" (UID: \"8cae7c13-8d21-49f1-b1d4-ea369ec4e54c\") " Nov 02 18:09:22.207287 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:22.207227 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cae7c13-8d21-49f1-b1d4-ea369ec4e54c-plugins-config-volume" pod "8cae7c13-8d21-49f1-b1d4-ea369ec4e54c" (UID: "8cae7c13-8d21-49f1-b1d4-ea369ec4e54c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:09:22.207819 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:22.207789 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cae7c13-8d21-49f1-b1d4-ea369ec4e54c-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:09:22.209146 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:22.209125 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cae7c13-8d21-49f1-b1d4-ea369ec4e54c-kube-api-access-ztvbn" pod "8cae7c13-8d21-49f1-b1d4-ea369ec4e54c" (UID: "8cae7c13-8d21-49f1-b1d4-ea369ec4e54c"). InnerVolumeSpecName "kube-api-access-ztvbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:09:22.309860 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:22.309820 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztvbn\" (UniqueName: \"kubernetes.io/projected/8cae7c13-8d21-49f1-b1d4-ea369ec4e54c-kube-api-access-ztvbn\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:09:22.443613 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:22.443539 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2" path="/var/lib/kubelet/pods/b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2/volumes" Nov 02 18:09:22.527451 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:22.527413 4471 generic.go:356] "Generic (PLEG): container finished" podID="8cae7c13-8d21-49f1-b1d4-ea369ec4e54c" containerID="7f61effc758f800483895725f323ac9c517c2158f2ac33abd5bd3cc3fa9a4bea" exitCode=0 Nov 02 18:09:22.527623 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:22.527465 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2f6787-6bb9757b5-tmdf8" Nov 02 18:09:22.527623 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:22.527475 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2f6787-6bb9757b5-tmdf8" event={"ID":"8cae7c13-8d21-49f1-b1d4-ea369ec4e54c","Type":"ContainerDied","Data":"7f61effc758f800483895725f323ac9c517c2158f2ac33abd5bd3cc3fa9a4bea"} Nov 02 18:09:22.527623 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:22.527526 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2f6787-6bb9757b5-tmdf8" event={"ID":"8cae7c13-8d21-49f1-b1d4-ea369ec4e54c","Type":"ContainerDied","Data":"b65e5a99f5dbbf22aa23c8e044855db5e11a3e37026975436878bbe7c684be21"} Nov 02 18:09:22.527623 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:22.527558 4471 scope.go:122] "RemoveContainer" containerID="7f61effc758f800483895725f323ac9c517c2158f2ac33abd5bd3cc3fa9a4bea" Nov 02 18:09:22.538380 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:22.538358 4471 scope.go:122] "RemoveContainer" containerID="7f61effc758f800483895725f323ac9c517c2158f2ac33abd5bd3cc3fa9a4bea" Nov 02 18:09:22.538665 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:09:22.538643 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f61effc758f800483895725f323ac9c517c2158f2ac33abd5bd3cc3fa9a4bea\": container with ID starting with 7f61effc758f800483895725f323ac9c517c2158f2ac33abd5bd3cc3fa9a4bea not found: ID does not exist" containerID="7f61effc758f800483895725f323ac9c517c2158f2ac33abd5bd3cc3fa9a4bea" Nov 02 18:09:22.538725 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:22.538672 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f61effc758f800483895725f323ac9c517c2158f2ac33abd5bd3cc3fa9a4bea"} err="failed to get container status \"7f61effc758f800483895725f323ac9c517c2158f2ac33abd5bd3cc3fa9a4bea\": rpc error: code = NotFound desc = could not find container \"7f61effc758f800483895725f323ac9c517c2158f2ac33abd5bd3cc3fa9a4bea\": container with ID starting with 7f61effc758f800483895725f323ac9c517c2158f2ac33abd5bd3cc3fa9a4bea not found: ID does not exist" Nov 02 18:09:22.591477 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:22.591441 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2f6787-6bb9757b5-tmdf8"] Nov 02 18:09:22.596457 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:22.596430 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2f6787-6bb9757b5-tmdf8"] Nov 02 18:09:23.130705 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:23.130656 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-2f6787-6bb9757b5-tmdf8" podUID="8cae7c13-8d21-49f1-b1d4-ea369ec4e54c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.137:50052\" within 1s: context deadline exceeded" Nov 02 18:09:23.130897 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:09:23.130744 4471 logging.go:55] [core] [Channel #8150 SubChannel #8151]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.137:50052", ServerName: "10.133.0.137:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.137:50052: operation was canceled" Nov 02 18:09:24.443116 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:24.443084 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cae7c13-8d21-49f1-b1d4-ea369ec4e54c" path="/var/lib/kubelet/pods/8cae7c13-8d21-49f1-b1d4-ea369ec4e54c/volumes" Nov 02 18:09:32.130895 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:32.130807 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6787-77f84bf7c9-bwsjs" podUID="721435f4-cdae-4071-9767-5dbf79c5dd3f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:09:34.128696 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:34.128635 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rt5cp_1b5ee10f-87a1-4168-904e-ba233f304fe3/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:39.185281 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:09:39.185237 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:09:42.228865 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:42.228836 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6787-77f84bf7c9-bwsjs" Nov 02 18:09:42.324558 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:42.324502 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/721435f4-cdae-4071-9767-5dbf79c5dd3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/721435f4-cdae-4071-9767-5dbf79c5dd3f-plugins-config-volume\") pod \"721435f4-cdae-4071-9767-5dbf79c5dd3f\" (UID: \"721435f4-cdae-4071-9767-5dbf79c5dd3f\") " Nov 02 18:09:42.325690 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:42.324676 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/721435f4-cdae-4071-9767-5dbf79c5dd3f-kube-api-access-bl6lr\" (UniqueName: \"kubernetes.io/projected/721435f4-cdae-4071-9767-5dbf79c5dd3f-kube-api-access-bl6lr\") pod \"721435f4-cdae-4071-9767-5dbf79c5dd3f\" (UID: \"721435f4-cdae-4071-9767-5dbf79c5dd3f\") " Nov 02 18:09:42.326088 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:42.326050 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/721435f4-cdae-4071-9767-5dbf79c5dd3f-plugins-config-volume" pod "721435f4-cdae-4071-9767-5dbf79c5dd3f" (UID: "721435f4-cdae-4071-9767-5dbf79c5dd3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:09:42.328052 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:42.327949 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/721435f4-cdae-4071-9767-5dbf79c5dd3f-kube-api-access-bl6lr" pod "721435f4-cdae-4071-9767-5dbf79c5dd3f" (UID: "721435f4-cdae-4071-9767-5dbf79c5dd3f"). InnerVolumeSpecName "kube-api-access-bl6lr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:09:42.427057 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:42.427024 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/721435f4-cdae-4071-9767-5dbf79c5dd3f-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:09:42.427057 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:42.427052 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl6lr\" (UniqueName: \"kubernetes.io/projected/721435f4-cdae-4071-9767-5dbf79c5dd3f-kube-api-access-bl6lr\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:09:42.718268 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:42.718179 4471 generic.go:356] "Generic (PLEG): container finished" podID="721435f4-cdae-4071-9767-5dbf79c5dd3f" containerID="623eafbab03185c9cb82308d3f856ff4f026ffd7af534c76eed53d3f5a130782" exitCode=0 Nov 02 18:09:42.718268 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:42.718247 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6787-77f84bf7c9-bwsjs" Nov 02 18:09:42.718268 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:42.718253 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6787-77f84bf7c9-bwsjs" event={"ID":"721435f4-cdae-4071-9767-5dbf79c5dd3f","Type":"ContainerDied","Data":"623eafbab03185c9cb82308d3f856ff4f026ffd7af534c76eed53d3f5a130782"} Nov 02 18:09:42.718526 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:42.718378 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6787-77f84bf7c9-bwsjs" event={"ID":"721435f4-cdae-4071-9767-5dbf79c5dd3f","Type":"ContainerDied","Data":"d9fe3e0a653395bea79f0431d70c509873c67ef2e1fb73985a5d608721d8e038"} Nov 02 18:09:42.718526 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:42.718412 4471 scope.go:122] "RemoveContainer" containerID="623eafbab03185c9cb82308d3f856ff4f026ffd7af534c76eed53d3f5a130782" Nov 02 18:09:42.729244 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:42.729222 4471 scope.go:122] "RemoveContainer" containerID="623eafbab03185c9cb82308d3f856ff4f026ffd7af534c76eed53d3f5a130782" Nov 02 18:09:42.729534 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:09:42.729512 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"623eafbab03185c9cb82308d3f856ff4f026ffd7af534c76eed53d3f5a130782\": container with ID starting with 623eafbab03185c9cb82308d3f856ff4f026ffd7af534c76eed53d3f5a130782 not found: ID does not exist" containerID="623eafbab03185c9cb82308d3f856ff4f026ffd7af534c76eed53d3f5a130782" Nov 02 18:09:42.729612 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:42.729542 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"623eafbab03185c9cb82308d3f856ff4f026ffd7af534c76eed53d3f5a130782"} err="failed to get container status \"623eafbab03185c9cb82308d3f856ff4f026ffd7af534c76eed53d3f5a130782\": rpc error: code = NotFound desc = could not find container \"623eafbab03185c9cb82308d3f856ff4f026ffd7af534c76eed53d3f5a130782\": container with ID starting with 623eafbab03185c9cb82308d3f856ff4f026ffd7af534c76eed53d3f5a130782 not found: ID does not exist" Nov 02 18:09:42.774711 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:42.774683 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2f6787-77f84bf7c9-bwsjs"] Nov 02 18:09:42.779341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:42.779315 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2f6787-77f84bf7c9-bwsjs"] Nov 02 18:09:43.130765 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:43.130722 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6787-77f84bf7c9-bwsjs" podUID="721435f4-cdae-4071-9767-5dbf79c5dd3f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.136:50052\" within 1s: context deadline exceeded" Nov 02 18:09:43.130967 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:09:43.130838 4471 logging.go:55] [core] [Channel #8269 SubChannel #8270]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.136:50052", ServerName: "10.133.0.136:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.136:50052: operation was canceled" Nov 02 18:09:44.443723 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:44.443689 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="721435f4-cdae-4071-9767-5dbf79c5dd3f" path="/var/lib/kubelet/pods/721435f4-cdae-4071-9767-5dbf79c5dd3f/volumes" Nov 02 18:09:45.701558 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:45.701519 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rt5cp_1b5ee10f-87a1-4168-904e-ba233f304fe3/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:51.684101 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:51.684063 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl"] Nov 02 18:09:51.837544 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:51.837505 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl" podUID="4cc197cd-1c89-46fc-974b-02d3bf536c3c" containerName="maas-api" containerID="cri-o://c68848ef4d90b0845b4728d4510abde0a824bd5cc2abbc4293ec04131cc43b61" gracePeriod=30 Nov 02 18:09:52.649255 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:52.649228 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl" Nov 02 18:09:52.661846 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:52.661822 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cc197cd-1c89-46fc-974b-02d3bf536c3c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cc197cd-1c89-46fc-974b-02d3bf536c3c-openshift-service-ca\") pod \"4cc197cd-1c89-46fc-974b-02d3bf536c3c\" (UID: \"4cc197cd-1c89-46fc-974b-02d3bf536c3c\") " Nov 02 18:09:52.661993 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:52.661927 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cc197cd-1c89-46fc-974b-02d3bf536c3c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cc197cd-1c89-46fc-974b-02d3bf536c3c-maas-api-tls\") pod \"4cc197cd-1c89-46fc-974b-02d3bf536c3c\" (UID: \"4cc197cd-1c89-46fc-974b-02d3bf536c3c\") " Nov 02 18:09:52.661993 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:52.661965 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cc197cd-1c89-46fc-974b-02d3bf536c3c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cc197cd-1c89-46fc-974b-02d3bf536c3c-metrics-certs\") pod \"4cc197cd-1c89-46fc-974b-02d3bf536c3c\" (UID: \"4cc197cd-1c89-46fc-974b-02d3bf536c3c\") " Nov 02 18:09:52.662225 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:52.662202 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cc197cd-1c89-46fc-974b-02d3bf536c3c-openshift-service-ca" pod "4cc197cd-1c89-46fc-974b-02d3bf536c3c" (UID: "4cc197cd-1c89-46fc-974b-02d3bf536c3c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:09:52.662278 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:52.662212 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cc197cd-1c89-46fc-974b-02d3bf536c3c-kube-api-access-rxknd\" (UniqueName: \"kubernetes.io/projected/4cc197cd-1c89-46fc-974b-02d3bf536c3c-kube-api-access-rxknd\") pod \"4cc197cd-1c89-46fc-974b-02d3bf536c3c\" (UID: \"4cc197cd-1c89-46fc-974b-02d3bf536c3c\") " Nov 02 18:09:52.662932 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:52.662908 4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cc197cd-1c89-46fc-974b-02d3bf536c3c-openshift-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:09:52.664014 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:52.663988 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cc197cd-1c89-46fc-974b-02d3bf536c3c-maas-api-tls" pod "4cc197cd-1c89-46fc-974b-02d3bf536c3c" (UID: "4cc197cd-1c89-46fc-974b-02d3bf536c3c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:09:52.664123 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:52.664106 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cc197cd-1c89-46fc-974b-02d3bf536c3c-metrics-certs" pod "4cc197cd-1c89-46fc-974b-02d3bf536c3c" (UID: "4cc197cd-1c89-46fc-974b-02d3bf536c3c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:09:52.664252 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:52.664231 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc197cd-1c89-46fc-974b-02d3bf536c3c-kube-api-access-rxknd" pod "4cc197cd-1c89-46fc-974b-02d3bf536c3c" (UID: "4cc197cd-1c89-46fc-974b-02d3bf536c3c"). InnerVolumeSpecName "kube-api-access-rxknd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:09:52.764151 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:52.764114 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxknd\" (UniqueName: \"kubernetes.io/projected/4cc197cd-1c89-46fc-974b-02d3bf536c3c-kube-api-access-rxknd\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:09:52.764151 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:52.764149 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cc197cd-1c89-46fc-974b-02d3bf536c3c-maas-api-tls\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:09:52.764759 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:52.764163 4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cc197cd-1c89-46fc-974b-02d3bf536c3c-metrics-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:09:52.907830 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:52.907729 4471 generic.go:356] "Generic (PLEG): container finished" podID="4cc197cd-1c89-46fc-974b-02d3bf536c3c" containerID="c68848ef4d90b0845b4728d4510abde0a824bd5cc2abbc4293ec04131cc43b61" exitCode=0 Nov 02 18:09:52.907830 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:52.907802 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl" Nov 02 18:09:52.908039 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:52.907801 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl" event={"ID":"4cc197cd-1c89-46fc-974b-02d3bf536c3c","Type":"ContainerDied","Data":"c68848ef4d90b0845b4728d4510abde0a824bd5cc2abbc4293ec04131cc43b61"} Nov 02 18:09:52.908039 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:52.907968 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl" event={"ID":"4cc197cd-1c89-46fc-974b-02d3bf536c3c","Type":"ContainerDied","Data":"458fe8f9dec0edea668767cab217a7aee313594119900cb5a14a329433899f2c"} Nov 02 18:09:52.908039 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:52.907999 4471 scope.go:122] "RemoveContainer" containerID="c68848ef4d90b0845b4728d4510abde0a824bd5cc2abbc4293ec04131cc43b61" Nov 02 18:09:52.922046 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:52.922021 4471 scope.go:122] "RemoveContainer" containerID="c68848ef4d90b0845b4728d4510abde0a824bd5cc2abbc4293ec04131cc43b61" Nov 02 18:09:52.922402 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:09:52.922385 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c68848ef4d90b0845b4728d4510abde0a824bd5cc2abbc4293ec04131cc43b61\": container with ID starting with c68848ef4d90b0845b4728d4510abde0a824bd5cc2abbc4293ec04131cc43b61 not found: ID does not exist" containerID="c68848ef4d90b0845b4728d4510abde0a824bd5cc2abbc4293ec04131cc43b61" Nov 02 18:09:52.922465 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:52.922411 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c68848ef4d90b0845b4728d4510abde0a824bd5cc2abbc4293ec04131cc43b61"} err="failed to get container status \"c68848ef4d90b0845b4728d4510abde0a824bd5cc2abbc4293ec04131cc43b61\": rpc error: code = NotFound desc = could not find container \"c68848ef4d90b0845b4728d4510abde0a824bd5cc2abbc4293ec04131cc43b61\": container with ID starting with c68848ef4d90b0845b4728d4510abde0a824bd5cc2abbc4293ec04131cc43b61 not found: ID does not exist" Nov 02 18:09:52.964535 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:52.964504 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl"] Nov 02 18:09:52.969163 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:52.969135 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6787-8cd5b5b58-64ttl"] Nov 02 18:09:54.444034 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:54.443999 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cc197cd-1c89-46fc-974b-02d3bf536c3c" path="/var/lib/kubelet/pods/4cc197cd-1c89-46fc-974b-02d3bf536c3c/volumes" Nov 02 18:09:55.888917 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:09:55.888878 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rt5cp_1b5ee10f-87a1-4168-904e-ba233f304fe3/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:02.893757 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:02.893720 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6787-gfmbn"] Nov 02 18:10:02.905924 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:02.905894 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6787-gfmbn"] Nov 02 18:10:04.443624 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:04.443583 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6" path="/var/lib/kubelet/pods/4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6/volumes" Nov 02 18:10:06.053751 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:06.053717 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rt5cp_1b5ee10f-87a1-4168-904e-ba233f304fe3/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:09.186542 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:10:09.186500 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:10:10.236545 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.236508 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4"] Nov 02 18:10:10.237448 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.237409 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" podUID="790f4f11-1fb0-49ae-871d-07295b20a266" containerName="istio-proxy" containerID="cri-o://b9e5afb1a653ef91c046b75fb888f50e24ac03adff234fed4762cbaab3baaf7f" gracePeriod=30 Nov 02 18:10:10.246135 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.246094 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597"] Nov 02 18:10:10.246425 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.246379 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" podUID="c1e9cee8-e915-4a1f-819d-226dcda7485a" containerName="istio-proxy" containerID="cri-o://18c0e4be2b5df625b39ee8e8cb2be8eb15ea519a0ee6d2ff9ff91e8f0c2dd0d8" gracePeriod=30 Nov 02 18:10:10.259061 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.258412 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg"] Nov 02 18:10:10.259061 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.258643 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" podUID="e304caf1-38c3-4cbb-9c2f-db257e76ada7" containerName="istio-proxy" containerID="cri-o://08ad3857b2a6d05b15a41996225a0a4807f64a50355e002fff80318bcf031a0f" gracePeriod=30 Nov 02 18:10:10.283604 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.267902 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c"] Nov 02 18:10:10.283604 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.268122 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" podUID="aaca053e-d726-46af-a659-2a15071343c7" containerName="istio-proxy" containerID="cri-o://d0b8e1501153337b07e1139059c008fa0659d99849d80e336abecb53916df430" gracePeriod=30 Nov 02 18:10:10.719550 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.719397 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2fc9f-ddbq6"] Nov 02 18:10:10.721905 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.721808 4471 cpu_manager.go:414] "containerMap: removing container" podUID="8cae7c13-8d21-49f1-b1d4-ea369ec4e54c" containerName="payload-processing" Nov 02 18:10:10.722799 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.722553 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cae7c13-8d21-49f1-b1d4-ea369ec4e54c" containerName="payload-processing" Nov 02 18:10:10.725688 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.725669 4471 cpu_manager.go:414] "containerMap: removing container" podUID="4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6" containerName="revoke-keys" Nov 02 18:10:10.725826 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.725811 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6" containerName="revoke-keys" Nov 02 18:10:10.725927 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.725916 4471 cpu_manager.go:414] "containerMap: removing container" podUID="721435f4-cdae-4071-9767-5dbf79c5dd3f" containerName="payload-pre-processing" Nov 02 18:10:10.726017 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.726005 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="721435f4-cdae-4071-9767-5dbf79c5dd3f" containerName="payload-pre-processing" Nov 02 18:10:10.726416 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.726400 4471 cpu_manager.go:414] "containerMap: removing container" podUID="4cc197cd-1c89-46fc-974b-02d3bf536c3c" containerName="maas-api" Nov 02 18:10:10.726536 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.726523 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cc197cd-1c89-46fc-974b-02d3bf536c3c" containerName="maas-api" Nov 02 18:10:10.726654 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.726642 4471 cpu_manager.go:414] "containerMap: removing container" podUID="b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2" containerName="main" Nov 02 18:10:10.726739 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.726729 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2" containerName="main" Nov 02 18:10:10.727081 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.727065 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cae7c13-8d21-49f1-b1d4-ea369ec4e54c" containerName="payload-processing" Nov 02 18:10:10.727193 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.727182 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="4172ebfa-094b-41bb-8e6a-9fde4ae2a7a6" containerName="revoke-keys" Nov 02 18:10:10.727280 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.727269 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="721435f4-cdae-4071-9767-5dbf79c5dd3f" containerName="payload-pre-processing" Nov 02 18:10:10.727404 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.727393 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cc197cd-1c89-46fc-974b-02d3bf536c3c" containerName="maas-api" Nov 02 18:10:10.727497 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.727487 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6aeceaa-bdb6-4f63-a0ea-f18cbc55bee2" containerName="main" Nov 02 18:10:10.731361 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.731345 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2fc9f-ddbq6"] Nov 02 18:10:10.731485 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.731455 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2fc9f-ddbq6" Nov 02 18:10:10.739545 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.739522 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl"] Nov 02 18:10:10.739982 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.739954 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" podUID="6183d0fa-f089-47f0-97a0-163160fc9f25" containerName="istio-proxy" containerID="cri-o://033e549a2214d5dc6f831beaf5d2f08b85a6941e8511c9022f60186592b9896a" gracePeriod=30 Nov 02 18:10:10.752533 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.751794 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 18:10:10.752533 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.752498 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:10:10.889403 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.888783 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c0f219b-fdac-4b79-92fc-80ae017a205a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b2fc9f-ddbq6\" (UID: \"4c0f219b-fdac-4b79-92fc-80ae017a205a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2fc9f-ddbq6" Nov 02 18:10:10.889403 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.889024 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c0f219b-fdac-4b79-92fc-80ae017a205a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b2fc9f-ddbq6\" (UID: \"4c0f219b-fdac-4b79-92fc-80ae017a205a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2fc9f-ddbq6" Nov 02 18:10:10.889403 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.889103 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl75z\" (UniqueName: \"kubernetes.io/projected/4c0f219b-fdac-4b79-92fc-80ae017a205a-kube-api-access-vl75z\") pod \"maas-api-revoke-keys-e2e-inf-b-b2fc9f-ddbq6\" (UID: \"4c0f219b-fdac-4b79-92fc-80ae017a205a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2fc9f-ddbq6" Nov 02 18:10:10.993378 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.992501 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl75z\" (UniqueName: \"kubernetes.io/projected/4c0f219b-fdac-4b79-92fc-80ae017a205a-kube-api-access-vl75z\") pod \"maas-api-revoke-keys-e2e-inf-b-b2fc9f-ddbq6\" (UID: \"4c0f219b-fdac-4b79-92fc-80ae017a205a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2fc9f-ddbq6" Nov 02 18:10:10.993378 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.992786 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c0f219b-fdac-4b79-92fc-80ae017a205a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b2fc9f-ddbq6\" (UID: \"4c0f219b-fdac-4b79-92fc-80ae017a205a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2fc9f-ddbq6" Nov 02 18:10:10.993378 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.992965 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c0f219b-fdac-4b79-92fc-80ae017a205a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b2fc9f-ddbq6\" (UID: \"4c0f219b-fdac-4b79-92fc-80ae017a205a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2fc9f-ddbq6" Nov 02 18:10:10.995334 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:10.994476 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c0f219b-fdac-4b79-92fc-80ae017a205a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b2fc9f-ddbq6\" (UID: \"4c0f219b-fdac-4b79-92fc-80ae017a205a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2fc9f-ddbq6" Nov 02 18:10:11.064379 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:11.062029 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl75z\" (UniqueName: \"kubernetes.io/projected/4c0f219b-fdac-4b79-92fc-80ae017a205a-kube-api-access-vl75z\") pod \"maas-api-revoke-keys-e2e-inf-b-b2fc9f-ddbq6\" (UID: \"4c0f219b-fdac-4b79-92fc-80ae017a205a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2fc9f-ddbq6" Nov 02 18:10:11.064379 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:11.062530 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c0f219b-fdac-4b79-92fc-80ae017a205a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b2fc9f-ddbq6\" (UID: \"4c0f219b-fdac-4b79-92fc-80ae017a205a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2fc9f-ddbq6" Nov 02 18:10:11.196356 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:11.196316 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5fd3e0-hkdvc"] Nov 02 18:10:11.200317 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:11.200287 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5fd3e0-hkdvc" Nov 02 18:10:11.218862 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:11.218827 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5fd3e0-hkdvc"] Nov 02 18:10:11.298497 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:11.298459 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a3d6e8-b54f-48ee-9381-4ee92f76e158-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5fd3e0-hkdvc\" (UID: \"f6a3d6e8-b54f-48ee-9381-4ee92f76e158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5fd3e0-hkdvc" Nov 02 18:10:11.298900 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:11.298593 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a3d6e8-b54f-48ee-9381-4ee92f76e158-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5fd3e0-hkdvc\" (UID: \"f6a3d6e8-b54f-48ee-9381-4ee92f76e158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5fd3e0-hkdvc" Nov 02 18:10:11.298900 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:11.298647 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvztg\" (UniqueName: \"kubernetes.io/projected/f6a3d6e8-b54f-48ee-9381-4ee92f76e158-kube-api-access-cvztg\") pod \"maas-api-revoke-keys-e2e-inf-b-5fd3e0-hkdvc\" (UID: \"f6a3d6e8-b54f-48ee-9381-4ee92f76e158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5fd3e0-hkdvc" Nov 02 18:10:11.352756 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:11.352719 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fbbbf8-7xz6v"] Nov 02 18:10:11.354683 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:11.354657 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2fc9f-ddbq6" Nov 02 18:10:11.356473 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:11.356451 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fbbbf8-7xz6v" Nov 02 18:10:11.378703 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:11.378677 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fbbbf8-7xz6v"] Nov 02 18:10:11.404327 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:11.401917 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a3d6e8-b54f-48ee-9381-4ee92f76e158-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5fd3e0-hkdvc\" (UID: \"f6a3d6e8-b54f-48ee-9381-4ee92f76e158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5fd3e0-hkdvc" Nov 02 18:10:11.404327 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:11.402064 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a3d6e8-b54f-48ee-9381-4ee92f76e158-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5fd3e0-hkdvc\" (UID: \"f6a3d6e8-b54f-48ee-9381-4ee92f76e158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5fd3e0-hkdvc" Nov 02 18:10:11.404327 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:11.402099 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvztg\" (UniqueName: \"kubernetes.io/projected/f6a3d6e8-b54f-48ee-9381-4ee92f76e158-kube-api-access-cvztg\") pod \"maas-api-revoke-keys-e2e-inf-b-5fd3e0-hkdvc\" (UID: \"f6a3d6e8-b54f-48ee-9381-4ee92f76e158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5fd3e0-hkdvc" Nov 02 18:10:11.404327 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:11.403553 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a3d6e8-b54f-48ee-9381-4ee92f76e158-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5fd3e0-hkdvc\" (UID: \"f6a3d6e8-b54f-48ee-9381-4ee92f76e158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5fd3e0-hkdvc" Nov 02 18:10:11.405727 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:11.405701 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a3d6e8-b54f-48ee-9381-4ee92f76e158-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5fd3e0-hkdvc\" (UID: \"f6a3d6e8-b54f-48ee-9381-4ee92f76e158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5fd3e0-hkdvc" Nov 02 18:10:11.450357 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:11.449992 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvztg\" (UniqueName: \"kubernetes.io/projected/f6a3d6e8-b54f-48ee-9381-4ee92f76e158-kube-api-access-cvztg\") pod \"maas-api-revoke-keys-e2e-inf-b-5fd3e0-hkdvc\" (UID: \"f6a3d6e8-b54f-48ee-9381-4ee92f76e158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5fd3e0-hkdvc" Nov 02 18:10:11.504928 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:11.504888 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a58f095-38b2-45e4-8cd0-1a2136540bbd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fbbbf8-7xz6v\" (UID: \"2a58f095-38b2-45e4-8cd0-1a2136540bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fbbbf8-7xz6v" Nov 02 18:10:11.505093 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:11.504947 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2ljb\" (UniqueName: \"kubernetes.io/projected/2a58f095-38b2-45e4-8cd0-1a2136540bbd-kube-api-access-m2ljb\") pod \"maas-api-revoke-keys-e2e-inf-b-fbbbf8-7xz6v\" (UID: \"2a58f095-38b2-45e4-8cd0-1a2136540bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fbbbf8-7xz6v" Nov 02 18:10:11.505185 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:11.505167 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a58f095-38b2-45e4-8cd0-1a2136540bbd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fbbbf8-7xz6v\" (UID: \"2a58f095-38b2-45e4-8cd0-1a2136540bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fbbbf8-7xz6v" Nov 02 18:10:11.519939 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:11.519905 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5fd3e0-hkdvc" Nov 02 18:10:11.610112 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:11.607733 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a58f095-38b2-45e4-8cd0-1a2136540bbd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fbbbf8-7xz6v\" (UID: \"2a58f095-38b2-45e4-8cd0-1a2136540bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fbbbf8-7xz6v" Nov 02 18:10:11.610112 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:11.607785 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2ljb\" (UniqueName: \"kubernetes.io/projected/2a58f095-38b2-45e4-8cd0-1a2136540bbd-kube-api-access-m2ljb\") pod \"maas-api-revoke-keys-e2e-inf-b-fbbbf8-7xz6v\" (UID: \"2a58f095-38b2-45e4-8cd0-1a2136540bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fbbbf8-7xz6v" Nov 02 18:10:11.610112 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:11.608081 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a58f095-38b2-45e4-8cd0-1a2136540bbd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fbbbf8-7xz6v\" (UID: \"2a58f095-38b2-45e4-8cd0-1a2136540bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fbbbf8-7xz6v" Nov 02 18:10:11.610112 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:11.609179 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a58f095-38b2-45e4-8cd0-1a2136540bbd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fbbbf8-7xz6v\" (UID: \"2a58f095-38b2-45e4-8cd0-1a2136540bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fbbbf8-7xz6v" Nov 02 18:10:11.611088 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:11.611064 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a58f095-38b2-45e4-8cd0-1a2136540bbd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fbbbf8-7xz6v\" (UID: \"2a58f095-38b2-45e4-8cd0-1a2136540bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fbbbf8-7xz6v" Nov 02 18:10:11.670696 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:11.670662 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2ljb\" (UniqueName: \"kubernetes.io/projected/2a58f095-38b2-45e4-8cd0-1a2136540bbd-kube-api-access-m2ljb\") pod \"maas-api-revoke-keys-e2e-inf-b-fbbbf8-7xz6v\" (UID: \"2a58f095-38b2-45e4-8cd0-1a2136540bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fbbbf8-7xz6v" Nov 02 18:10:11.705878 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:10:11.705846 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c0f219b_fdac_4b79_92fc_80ae017a205a.slice/crio-daeb1bfddceda66a97acc5816533cc3d551bdf2ed61daf8e536bb57ec66e1087 WatchSource:0}: Error finding container daeb1bfddceda66a97acc5816533cc3d551bdf2ed61daf8e536bb57ec66e1087: Status 404 returned error can't find the container with id daeb1bfddceda66a97acc5816533cc3d551bdf2ed61daf8e536bb57ec66e1087 Nov 02 18:10:11.707501 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:11.707473 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2fc9f-ddbq6"] Nov 02 18:10:11.717568 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:11.717544 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fbbbf8-7xz6v" Nov 02 18:10:11.838049 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:11.838013 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5fd3e0-hkdvc"] Nov 02 18:10:11.859616 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:10:11.859583 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a3d6e8_b54f_48ee_9381_4ee92f76e158.slice/crio-f050b8c8501aad588314c5b835309af913686498f7c64cc9df95a142baec420e WatchSource:0}: Error finding container f050b8c8501aad588314c5b835309af913686498f7c64cc9df95a142baec420e: Status 404 returned error can't find the container with id f050b8c8501aad588314c5b835309af913686498f7c64cc9df95a142baec420e Nov 02 18:10:12.046006 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:12.045977 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fbbbf8-7xz6v"] Nov 02 18:10:12.153720 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:12.153685 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fbbbf8-7xz6v" event={"ID":"2a58f095-38b2-45e4-8cd0-1a2136540bbd","Type":"ContainerStarted","Data":"326305de8c75bcf318c0ddd6f3848810b5b2f15802a10a413c14cbfb284285c2"} Nov 02 18:10:12.155179 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:12.155155 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2fc9f-ddbq6" event={"ID":"4c0f219b-fdac-4b79-92fc-80ae017a205a","Type":"ContainerStarted","Data":"daeb1bfddceda66a97acc5816533cc3d551bdf2ed61daf8e536bb57ec66e1087"} Nov 02 18:10:12.156787 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:12.156759 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5fd3e0-hkdvc" event={"ID":"f6a3d6e8-b54f-48ee-9381-4ee92f76e158","Type":"ContainerStarted","Data":"f050b8c8501aad588314c5b835309af913686498f7c64cc9df95a142baec420e"} Nov 02 18:10:13.116380 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:13.116241 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs"] Nov 02 18:10:13.116840 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:13.116547 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" podUID="19632b35-0e41-4d90-89c3-ced789733fbc" containerName="istio-proxy" containerID="cri-o://f2aae250a184d3f9120231114c61a5f8aceebfd1e61d53603be12066a16e1b01" gracePeriod=30 Nov 02 18:10:13.166953 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:13.166922 4471 generic.go:356] "Generic (PLEG): container finished" podID="4c0f219b-fdac-4b79-92fc-80ae017a205a" containerID="32cdac7d1038d4b611349439e522bb34a33f2270bf4dd52f06417698debb4715" exitCode=0 Nov 02 18:10:13.167123 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:13.167064 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2fc9f-ddbq6" event={"ID":"4c0f219b-fdac-4b79-92fc-80ae017a205a","Type":"ContainerDied","Data":"32cdac7d1038d4b611349439e522bb34a33f2270bf4dd52f06417698debb4715"} Nov 02 18:10:13.168888 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:13.168865 4471 generic.go:356] "Generic (PLEG): container finished" podID="f6a3d6e8-b54f-48ee-9381-4ee92f76e158" containerID="52ba382858dfe2119909f86972e5a52f2b1bada1c087726856256cd22fafb54a" exitCode=0 Nov 02 18:10:13.169025 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:13.169010 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5fd3e0-hkdvc" event={"ID":"f6a3d6e8-b54f-48ee-9381-4ee92f76e158","Type":"ContainerDied","Data":"52ba382858dfe2119909f86972e5a52f2b1bada1c087726856256cd22fafb54a"} Nov 02 18:10:13.170913 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:13.170892 4471 generic.go:356] "Generic (PLEG): container finished" podID="2a58f095-38b2-45e4-8cd0-1a2136540bbd" containerID="0a963746b0cc61037f2ab752432415c0f223c1f491af9aeb2295e6678ff7ee58" exitCode=0 Nov 02 18:10:13.170991 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:13.170921 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fbbbf8-7xz6v" event={"ID":"2a58f095-38b2-45e4-8cd0-1a2136540bbd","Type":"ContainerDied","Data":"0a963746b0cc61037f2ab752432415c0f223c1f491af9aeb2295e6678ff7ee58"} Nov 02 18:10:13.735043 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:13.735000 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5b905-fvz8z"] Nov 02 18:10:13.739254 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:13.739226 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5b905-fvz8z" Nov 02 18:10:13.758717 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:13.758688 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5b905-fvz8z"] Nov 02 18:10:13.786617 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:13.786570 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5flc7\" (UniqueName: \"kubernetes.io/projected/0b34a701-da68-40d6-9bd1-bcdd3a3e2866-kube-api-access-5flc7\") pod \"maas-api-revoke-keys-e2e-inf-b-d5b905-fvz8z\" (UID: \"0b34a701-da68-40d6-9bd1-bcdd3a3e2866\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5b905-fvz8z" Nov 02 18:10:13.786617 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:13.786620 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b34a701-da68-40d6-9bd1-bcdd3a3e2866-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d5b905-fvz8z\" (UID: \"0b34a701-da68-40d6-9bd1-bcdd3a3e2866\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5b905-fvz8z" Nov 02 18:10:13.786866 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:13.786689 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b34a701-da68-40d6-9bd1-bcdd3a3e2866-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d5b905-fvz8z\" (UID: \"0b34a701-da68-40d6-9bd1-bcdd3a3e2866\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5b905-fvz8z" Nov 02 18:10:13.888410 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:13.888375 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5flc7\" (UniqueName: \"kubernetes.io/projected/0b34a701-da68-40d6-9bd1-bcdd3a3e2866-kube-api-access-5flc7\") pod \"maas-api-revoke-keys-e2e-inf-b-d5b905-fvz8z\" (UID: \"0b34a701-da68-40d6-9bd1-bcdd3a3e2866\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5b905-fvz8z" Nov 02 18:10:13.888583 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:13.888430 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b34a701-da68-40d6-9bd1-bcdd3a3e2866-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d5b905-fvz8z\" (UID: \"0b34a701-da68-40d6-9bd1-bcdd3a3e2866\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5b905-fvz8z" Nov 02 18:10:13.888583 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:13.888461 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b34a701-da68-40d6-9bd1-bcdd3a3e2866-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d5b905-fvz8z\" (UID: \"0b34a701-da68-40d6-9bd1-bcdd3a3e2866\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5b905-fvz8z" Nov 02 18:10:13.889151 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:13.889131 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b34a701-da68-40d6-9bd1-bcdd3a3e2866-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d5b905-fvz8z\" (UID: \"0b34a701-da68-40d6-9bd1-bcdd3a3e2866\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5b905-fvz8z" Nov 02 18:10:13.891220 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:13.891194 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b34a701-da68-40d6-9bd1-bcdd3a3e2866-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d5b905-fvz8z\" (UID: \"0b34a701-da68-40d6-9bd1-bcdd3a3e2866\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5b905-fvz8z" Nov 02 18:10:13.913057 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:13.913030 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5flc7\" (UniqueName: \"kubernetes.io/projected/0b34a701-da68-40d6-9bd1-bcdd3a3e2866-kube-api-access-5flc7\") pod \"maas-api-revoke-keys-e2e-inf-b-d5b905-fvz8z\" (UID: \"0b34a701-da68-40d6-9bd1-bcdd3a3e2866\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5b905-fvz8z" Nov 02 18:10:14.066000 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:14.065027 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5b905-fvz8z" Nov 02 18:10:14.467836 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:14.462439 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5b905-fvz8z"] Nov 02 18:10:14.547194 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:14.547167 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2fc9f-ddbq6" Nov 02 18:10:14.550872 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:14.550850 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fbbbf8-7xz6v" Nov 02 18:10:14.554912 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:14.554891 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5fd3e0-hkdvc" Nov 02 18:10:14.626273 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:14.626184 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a58f095-38b2-45e4-8cd0-1a2136540bbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a58f095-38b2-45e4-8cd0-1a2136540bbd-maas-api-xks-ca\") pod \"2a58f095-38b2-45e4-8cd0-1a2136540bbd\" (UID: \"2a58f095-38b2-45e4-8cd0-1a2136540bbd\") " Nov 02 18:10:14.626273 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:14.626253 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c0f219b-fdac-4b79-92fc-80ae017a205a-kube-api-access-vl75z\" (UniqueName: \"kubernetes.io/projected/4c0f219b-fdac-4b79-92fc-80ae017a205a-kube-api-access-vl75z\") pod \"4c0f219b-fdac-4b79-92fc-80ae017a205a\" (UID: \"4c0f219b-fdac-4b79-92fc-80ae017a205a\") " Nov 02 18:10:14.626530 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:14.626369 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6a3d6e8-b54f-48ee-9381-4ee92f76e158-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a3d6e8-b54f-48ee-9381-4ee92f76e158-maas-api-openshift-ca\") pod \"f6a3d6e8-b54f-48ee-9381-4ee92f76e158\" (UID: \"f6a3d6e8-b54f-48ee-9381-4ee92f76e158\") " Nov 02 18:10:14.626530 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:14.626445 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a58f095-38b2-45e4-8cd0-1a2136540bbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a58f095-38b2-45e4-8cd0-1a2136540bbd-maas-api-openshift-ca\") pod \"2a58f095-38b2-45e4-8cd0-1a2136540bbd\" (UID: \"2a58f095-38b2-45e4-8cd0-1a2136540bbd\") " Nov 02 18:10:14.626628 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:14.626604 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6a3d6e8-b54f-48ee-9381-4ee92f76e158-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a3d6e8-b54f-48ee-9381-4ee92f76e158-maas-api-xks-ca\") pod \"f6a3d6e8-b54f-48ee-9381-4ee92f76e158\" (UID: \"f6a3d6e8-b54f-48ee-9381-4ee92f76e158\") " Nov 02 18:10:14.626688 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:14.626658 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6a3d6e8-b54f-48ee-9381-4ee92f76e158-kube-api-access-cvztg\" (UniqueName: \"kubernetes.io/projected/f6a3d6e8-b54f-48ee-9381-4ee92f76e158-kube-api-access-cvztg\") pod \"f6a3d6e8-b54f-48ee-9381-4ee92f76e158\" (UID: \"f6a3d6e8-b54f-48ee-9381-4ee92f76e158\") " Nov 02 18:10:14.626801 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:14.626781 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a58f095-38b2-45e4-8cd0-1a2136540bbd-kube-api-access-m2ljb\" (UniqueName: \"kubernetes.io/projected/2a58f095-38b2-45e4-8cd0-1a2136540bbd-kube-api-access-m2ljb\") pod \"2a58f095-38b2-45e4-8cd0-1a2136540bbd\" (UID: \"2a58f095-38b2-45e4-8cd0-1a2136540bbd\") " Nov 02 18:10:14.626939 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:14.626854 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c0f219b-fdac-4b79-92fc-80ae017a205a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c0f219b-fdac-4b79-92fc-80ae017a205a-maas-api-openshift-ca\") pod \"4c0f219b-fdac-4b79-92fc-80ae017a205a\" (UID: \"4c0f219b-fdac-4b79-92fc-80ae017a205a\") " Nov 02 18:10:14.627062 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:14.627042 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c0f219b-fdac-4b79-92fc-80ae017a205a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c0f219b-fdac-4b79-92fc-80ae017a205a-maas-api-xks-ca\") pod \"4c0f219b-fdac-4b79-92fc-80ae017a205a\" (UID: \"4c0f219b-fdac-4b79-92fc-80ae017a205a\") " Nov 02 18:10:14.628688 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:14.627045 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6a3d6e8-b54f-48ee-9381-4ee92f76e158-maas-api-openshift-ca" pod "f6a3d6e8-b54f-48ee-9381-4ee92f76e158" (UID: "f6a3d6e8-b54f-48ee-9381-4ee92f76e158"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:14.628688 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:14.627743 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a58f095-38b2-45e4-8cd0-1a2136540bbd-maas-api-openshift-ca" pod "2a58f095-38b2-45e4-8cd0-1a2136540bbd" (UID: "2a58f095-38b2-45e4-8cd0-1a2136540bbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:14.628688 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:14.628062 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c0f219b-fdac-4b79-92fc-80ae017a205a-maas-api-openshift-ca" pod "4c0f219b-fdac-4b79-92fc-80ae017a205a" (UID: "4c0f219b-fdac-4b79-92fc-80ae017a205a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:14.631002 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:14.629535 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c0f219b-fdac-4b79-92fc-80ae017a205a-kube-api-access-vl75z" pod "4c0f219b-fdac-4b79-92fc-80ae017a205a" (UID: "4c0f219b-fdac-4b79-92fc-80ae017a205a"). InnerVolumeSpecName "kube-api-access-vl75z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:14.631002 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:14.630718 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a58f095-38b2-45e4-8cd0-1a2136540bbd-maas-api-xks-ca" pod "2a58f095-38b2-45e4-8cd0-1a2136540bbd" (UID: "2a58f095-38b2-45e4-8cd0-1a2136540bbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:10:14.631720 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:14.631627 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a3d6e8-b54f-48ee-9381-4ee92f76e158-kube-api-access-cvztg" pod "f6a3d6e8-b54f-48ee-9381-4ee92f76e158" (UID: "f6a3d6e8-b54f-48ee-9381-4ee92f76e158"). InnerVolumeSpecName "kube-api-access-cvztg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:14.633458 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:14.633428 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c0f219b-fdac-4b79-92fc-80ae017a205a-maas-api-xks-ca" pod "4c0f219b-fdac-4b79-92fc-80ae017a205a" (UID: "4c0f219b-fdac-4b79-92fc-80ae017a205a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:10:14.633668 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:14.633472 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a3d6e8-b54f-48ee-9381-4ee92f76e158-maas-api-xks-ca" pod "f6a3d6e8-b54f-48ee-9381-4ee92f76e158" (UID: "f6a3d6e8-b54f-48ee-9381-4ee92f76e158"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:10:14.633668 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:14.633504 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a58f095-38b2-45e4-8cd0-1a2136540bbd-kube-api-access-m2ljb" pod "2a58f095-38b2-45e4-8cd0-1a2136540bbd" (UID: "2a58f095-38b2-45e4-8cd0-1a2136540bbd"). InnerVolumeSpecName "kube-api-access-m2ljb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:14.730457 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:14.730418 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a3d6e8-b54f-48ee-9381-4ee92f76e158-maas-api-xks-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:14.730457 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:14.730458 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvztg\" (UniqueName: \"kubernetes.io/projected/f6a3d6e8-b54f-48ee-9381-4ee92f76e158-kube-api-access-cvztg\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:14.730670 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:14.730476 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2ljb\" (UniqueName: \"kubernetes.io/projected/2a58f095-38b2-45e4-8cd0-1a2136540bbd-kube-api-access-m2ljb\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:14.730670 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:14.730491 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c0f219b-fdac-4b79-92fc-80ae017a205a-maas-api-openshift-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:14.730670 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:14.730506 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c0f219b-fdac-4b79-92fc-80ae017a205a-maas-api-xks-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:14.730670 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:14.730521 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a58f095-38b2-45e4-8cd0-1a2136540bbd-maas-api-xks-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:14.730670 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:14.730535 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl75z\" (UniqueName: \"kubernetes.io/projected/4c0f219b-fdac-4b79-92fc-80ae017a205a-kube-api-access-vl75z\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:14.730670 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:14.730551 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a3d6e8-b54f-48ee-9381-4ee92f76e158-maas-api-openshift-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:14.730670 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:14.730564 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a58f095-38b2-45e4-8cd0-1a2136540bbd-maas-api-openshift-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.192500 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.192469 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fbbbf8-7xz6v" Nov 02 18:10:15.192685 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.192491 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fbbbf8-7xz6v" event={"ID":"2a58f095-38b2-45e4-8cd0-1a2136540bbd","Type":"ContainerDied","Data":"326305de8c75bcf318c0ddd6f3848810b5b2f15802a10a413c14cbfb284285c2"} Nov 02 18:10:15.192685 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.192550 4471 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="326305de8c75bcf318c0ddd6f3848810b5b2f15802a10a413c14cbfb284285c2" Nov 02 18:10:15.194240 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.194217 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5b905-fvz8z" event={"ID":"0b34a701-da68-40d6-9bd1-bcdd3a3e2866","Type":"ContainerStarted","Data":"1a022d508a380be3b9103abe792e805d2c3f24a8ee8ba1689b731cad78cfb6fa"} Nov 02 18:10:15.194418 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.194259 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5b905-fvz8z" event={"ID":"0b34a701-da68-40d6-9bd1-bcdd3a3e2866","Type":"ContainerStarted","Data":"96e6f5917a205fdbae30e0beb8f915bbd0bb3450ecf00c9fc5a38df76f1d903b"} Nov 02 18:10:15.195672 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.195646 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2fc9f-ddbq6" Nov 02 18:10:15.195764 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.195685 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2fc9f-ddbq6" event={"ID":"4c0f219b-fdac-4b79-92fc-80ae017a205a","Type":"ContainerDied","Data":"daeb1bfddceda66a97acc5816533cc3d551bdf2ed61daf8e536bb57ec66e1087"} Nov 02 18:10:15.195764 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.195731 4471 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="daeb1bfddceda66a97acc5816533cc3d551bdf2ed61daf8e536bb57ec66e1087" Nov 02 18:10:15.197336 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.197286 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5fd3e0-hkdvc" event={"ID":"f6a3d6e8-b54f-48ee-9381-4ee92f76e158","Type":"ContainerDied","Data":"f050b8c8501aad588314c5b835309af913686498f7c64cc9df95a142baec420e"} Nov 02 18:10:15.197424 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.197357 4471 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f050b8c8501aad588314c5b835309af913686498f7c64cc9df95a142baec420e" Nov 02 18:10:15.197424 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.197320 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5fd3e0-hkdvc" Nov 02 18:10:15.238527 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.238479 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5b905-fvz8z" podStartSLOduration=2.238466039 podStartE2EDuration="2.238466039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:10:15.236510909 +0000 UTC m=+2432.931866402" watchObservedRunningTime="2026-11-02 18:10:15.238466039 +0000 UTC m=+2432.933821516" Nov 02 18:10:15.589211 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.589182 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 18:10:15.634660 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.634635 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 18:10:15.649045 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.648987 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-istio-data\") pod \"c1e9cee8-e915-4a1f-819d-226dcda7485a\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " Nov 02 18:10:15.649351 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.649323 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1e9cee8-e915-4a1f-819d-226dcda7485a-istio-token\" (UniqueName: \"kubernetes.io/projected/c1e9cee8-e915-4a1f-819d-226dcda7485a-istio-token\") pod \"c1e9cee8-e915-4a1f-819d-226dcda7485a\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " Nov 02 18:10:15.649585 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.649542 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-credential-socket\") pod \"c1e9cee8-e915-4a1f-819d-226dcda7485a\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " Nov 02 18:10:15.650075 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.650055 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-istio-envoy\") pod \"c1e9cee8-e915-4a1f-819d-226dcda7485a\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " Nov 02 18:10:15.650232 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.650215 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1e9cee8-e915-4a1f-819d-226dcda7485a-kube-api-access-bwp8r\" (UniqueName: \"kubernetes.io/projected/c1e9cee8-e915-4a1f-819d-226dcda7485a-kube-api-access-bwp8r\") pod \"c1e9cee8-e915-4a1f-819d-226dcda7485a\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " Nov 02 18:10:15.650395 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.650381 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1e9cee8-e915-4a1f-819d-226dcda7485a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e9cee8-e915-4a1f-819d-226dcda7485a-istio-podinfo\") pod \"c1e9cee8-e915-4a1f-819d-226dcda7485a\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " Nov 02 18:10:15.650554 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.650540 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-workload-certs\") pod \"c1e9cee8-e915-4a1f-819d-226dcda7485a\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " Nov 02 18:10:15.650821 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.650802 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-workload-socket\") pod \"c1e9cee8-e915-4a1f-819d-226dcda7485a\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " Nov 02 18:10:15.651027 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.651011 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1e9cee8-e915-4a1f-819d-226dcda7485a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e9cee8-e915-4a1f-819d-226dcda7485a-istiod-ca-cert\") pod \"c1e9cee8-e915-4a1f-819d-226dcda7485a\" (UID: \"c1e9cee8-e915-4a1f-819d-226dcda7485a\") " Nov 02 18:10:15.651769 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.649359 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-istio-data" pod "c1e9cee8-e915-4a1f-819d-226dcda7485a" (UID: "c1e9cee8-e915-4a1f-819d-226dcda7485a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.652074 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.650121 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-credential-socket" pod "c1e9cee8-e915-4a1f-819d-226dcda7485a" (UID: "c1e9cee8-e915-4a1f-819d-226dcda7485a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.652074 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.651352 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-workload-socket" pod "c1e9cee8-e915-4a1f-819d-226dcda7485a" (UID: "c1e9cee8-e915-4a1f-819d-226dcda7485a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.652074 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.651406 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-workload-certs" pod "c1e9cee8-e915-4a1f-819d-226dcda7485a" (UID: "c1e9cee8-e915-4a1f-819d-226dcda7485a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.652074 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.651725 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1e9cee8-e915-4a1f-819d-226dcda7485a-istiod-ca-cert" pod "c1e9cee8-e915-4a1f-819d-226dcda7485a" (UID: "c1e9cee8-e915-4a1f-819d-226dcda7485a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:15.652360 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.652266 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.652360 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.652289 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.652360 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.652321 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.652360 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.652337 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.652360 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.652349 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e9cee8-e915-4a1f-819d-226dcda7485a-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.658775 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.658569 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e9cee8-e915-4a1f-819d-226dcda7485a-kube-api-access-bwp8r" pod "c1e9cee8-e915-4a1f-819d-226dcda7485a" (UID: "c1e9cee8-e915-4a1f-819d-226dcda7485a"). InnerVolumeSpecName "kube-api-access-bwp8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:15.658775 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.658712 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e9cee8-e915-4a1f-819d-226dcda7485a-istio-token" pod "c1e9cee8-e915-4a1f-819d-226dcda7485a" (UID: "c1e9cee8-e915-4a1f-819d-226dcda7485a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:15.659247 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.659226 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-istio-envoy" pod "c1e9cee8-e915-4a1f-819d-226dcda7485a" (UID: "c1e9cee8-e915-4a1f-819d-226dcda7485a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.660524 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.660484 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1e9cee8-e915-4a1f-819d-226dcda7485a-istio-podinfo" pod "c1e9cee8-e915-4a1f-819d-226dcda7485a" (UID: "c1e9cee8-e915-4a1f-819d-226dcda7485a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:10:15.739813 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.739783 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 18:10:15.743472 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.743454 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:10:15.753155 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.753128 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/790f4f11-1fb0-49ae-871d-07295b20a266-istio-token\" (UniqueName: \"kubernetes.io/projected/790f4f11-1fb0-49ae-871d-07295b20a266-istio-token\") pod \"790f4f11-1fb0-49ae-871d-07295b20a266\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " Nov 02 18:10:15.753276 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.753196 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-workload-certs\") pod \"790f4f11-1fb0-49ae-871d-07295b20a266\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " Nov 02 18:10:15.753276 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.753250 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-istio-data\") pod \"790f4f11-1fb0-49ae-871d-07295b20a266\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " Nov 02 18:10:15.753492 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.753446 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-workload-certs" pod "790f4f11-1fb0-49ae-871d-07295b20a266" (UID: "790f4f11-1fb0-49ae-871d-07295b20a266"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.753560 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.753517 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-istio-data" pod "790f4f11-1fb0-49ae-871d-07295b20a266" (UID: "790f4f11-1fb0-49ae-871d-07295b20a266"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.753560 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.753542 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-istio-envoy\") pod \"790f4f11-1fb0-49ae-871d-07295b20a266\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " Nov 02 18:10:15.753683 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.753595 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-workload-socket\") pod \"790f4f11-1fb0-49ae-871d-07295b20a266\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " Nov 02 18:10:15.753759 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.753732 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-credential-socket\") pod \"790f4f11-1fb0-49ae-871d-07295b20a266\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " Nov 02 18:10:15.753822 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.753774 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-workload-socket" pod "790f4f11-1fb0-49ae-871d-07295b20a266" (UID: "790f4f11-1fb0-49ae-871d-07295b20a266"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.753822 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.753806 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/790f4f11-1fb0-49ae-871d-07295b20a266-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790f4f11-1fb0-49ae-871d-07295b20a266-istio-podinfo\") pod \"790f4f11-1fb0-49ae-871d-07295b20a266\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " Nov 02 18:10:15.753953 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.753935 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-credential-socket" pod "790f4f11-1fb0-49ae-871d-07295b20a266" (UID: "790f4f11-1fb0-49ae-871d-07295b20a266"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.754017 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.753984 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/790f4f11-1fb0-49ae-871d-07295b20a266-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790f4f11-1fb0-49ae-871d-07295b20a266-istiod-ca-cert\") pod \"790f4f11-1fb0-49ae-871d-07295b20a266\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " Nov 02 18:10:15.754078 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.754048 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/790f4f11-1fb0-49ae-871d-07295b20a266-kube-api-access-vb6z7\" (UniqueName: \"kubernetes.io/projected/790f4f11-1fb0-49ae-871d-07295b20a266-kube-api-access-vb6z7\") pod \"790f4f11-1fb0-49ae-871d-07295b20a266\" (UID: \"790f4f11-1fb0-49ae-871d-07295b20a266\") " Nov 02 18:10:15.754266 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.754243 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/790f4f11-1fb0-49ae-871d-07295b20a266-istiod-ca-cert" pod "790f4f11-1fb0-49ae-871d-07295b20a266" (UID: "790f4f11-1fb0-49ae-871d-07295b20a266"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:15.755450 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.755423 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.755553 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.755456 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e9cee8-e915-4a1f-819d-226dcda7485a-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.755553 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.755472 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwp8r\" (UniqueName: \"kubernetes.io/projected/c1e9cee8-e915-4a1f-819d-226dcda7485a-kube-api-access-bwp8r\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.755553 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.755488 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e9cee8-e915-4a1f-819d-226dcda7485a-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.755553 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.755504 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.755553 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.755520 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790f4f11-1fb0-49ae-871d-07295b20a266-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.755553 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.755535 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.755553 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.755547 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.756001 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.755560 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e9cee8-e915-4a1f-819d-226dcda7485a-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.756239 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.756214 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/790f4f11-1fb0-49ae-871d-07295b20a266-istio-podinfo" pod "790f4f11-1fb0-49ae-871d-07295b20a266" (UID: "790f4f11-1fb0-49ae-871d-07295b20a266"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:10:15.756239 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.756233 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-istio-envoy" pod "790f4f11-1fb0-49ae-871d-07295b20a266" (UID: "790f4f11-1fb0-49ae-871d-07295b20a266"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.758432 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.757977 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/790f4f11-1fb0-49ae-871d-07295b20a266-istio-token" pod "790f4f11-1fb0-49ae-871d-07295b20a266" (UID: "790f4f11-1fb0-49ae-871d-07295b20a266"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:15.758432 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.758047 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/790f4f11-1fb0-49ae-871d-07295b20a266-kube-api-access-vb6z7" pod "790f4f11-1fb0-49ae-871d-07295b20a266" (UID: "790f4f11-1fb0-49ae-871d-07295b20a266"). InnerVolumeSpecName "kube-api-access-vb6z7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:15.857968 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.856964 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-istio-data\") pod \"aaca053e-d726-46af-a659-2a15071343c7\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " Nov 02 18:10:15.857968 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.857018 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istio-podinfo\") pod \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " Nov 02 18:10:15.857968 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.857052 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-credential-socket\") pod \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " Nov 02 18:10:15.857968 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.857138 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istio-token\" (UniqueName: \"kubernetes.io/projected/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istio-token\") pod \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " Nov 02 18:10:15.857968 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.857182 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e304caf1-38c3-4cbb-9c2f-db257e76ada7-kube-api-access-l458b\" (UniqueName: \"kubernetes.io/projected/e304caf1-38c3-4cbb-9c2f-db257e76ada7-kube-api-access-l458b\") pod \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " Nov 02 18:10:15.857968 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.857211 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-workload-socket\") pod \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " Nov 02 18:10:15.857968 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.857297 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-credential-socket\") pod \"aaca053e-d726-46af-a659-2a15071343c7\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " Nov 02 18:10:15.857968 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.857349 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istio-data\") pod \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " Nov 02 18:10:15.857968 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.857388 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaca053e-d726-46af-a659-2a15071343c7-istio-token\" (UniqueName: \"kubernetes.io/projected/aaca053e-d726-46af-a659-2a15071343c7-istio-token\") pod \"aaca053e-d726-46af-a659-2a15071343c7\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " Nov 02 18:10:15.857968 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.857491 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-workload-socket\") pod \"aaca053e-d726-46af-a659-2a15071343c7\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " Nov 02 18:10:15.857968 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.857519 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-istio-envoy\") pod \"aaca053e-d726-46af-a659-2a15071343c7\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " Nov 02 18:10:15.857968 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.857568 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-workload-certs\") pod \"aaca053e-d726-46af-a659-2a15071343c7\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " Nov 02 18:10:15.857968 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.857595 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aaca053e-d726-46af-a659-2a15071343c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaca053e-d726-46af-a659-2a15071343c7-istio-podinfo\") pod \"aaca053e-d726-46af-a659-2a15071343c7\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " Nov 02 18:10:15.857968 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.857706 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istio-envoy\") pod \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " Nov 02 18:10:15.857968 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.857737 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-workload-certs\") pod \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " Nov 02 18:10:15.857968 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.857772 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaca053e-d726-46af-a659-2a15071343c7-kube-api-access-6rgpz\" (UniqueName: \"kubernetes.io/projected/aaca053e-d726-46af-a659-2a15071343c7-kube-api-access-6rgpz\") pod \"aaca053e-d726-46af-a659-2a15071343c7\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " Nov 02 18:10:15.857968 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.857808 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istiod-ca-cert\") pod \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\" (UID: \"e304caf1-38c3-4cbb-9c2f-db257e76ada7\") " Nov 02 18:10:15.857968 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.857855 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaca053e-d726-46af-a659-2a15071343c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaca053e-d726-46af-a659-2a15071343c7-istiod-ca-cert\") pod \"aaca053e-d726-46af-a659-2a15071343c7\" (UID: \"aaca053e-d726-46af-a659-2a15071343c7\") " Nov 02 18:10:15.859222 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.858848 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790f4f11-1fb0-49ae-871d-07295b20a266-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.859222 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.858875 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb6z7\" (UniqueName: \"kubernetes.io/projected/790f4f11-1fb0-49ae-871d-07295b20a266-kube-api-access-vb6z7\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.859222 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.858890 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/790f4f11-1fb0-49ae-871d-07295b20a266-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.859222 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.858903 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790f4f11-1fb0-49ae-871d-07295b20a266-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.859492 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.859400 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-credential-socket" pod "e304caf1-38c3-4cbb-9c2f-db257e76ada7" (UID: "e304caf1-38c3-4cbb-9c2f-db257e76ada7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.860838 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.860236 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istio-podinfo" pod "e304caf1-38c3-4cbb-9c2f-db257e76ada7" (UID: "e304caf1-38c3-4cbb-9c2f-db257e76ada7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:10:15.860838 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.860438 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaca053e-d726-46af-a659-2a15071343c7-istiod-ca-cert" pod "aaca053e-d726-46af-a659-2a15071343c7" (UID: "aaca053e-d726-46af-a659-2a15071343c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:15.860838 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.860570 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-istio-data" pod "aaca053e-d726-46af-a659-2a15071343c7" (UID: "aaca053e-d726-46af-a659-2a15071343c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.861492 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.861465 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-credential-socket" pod "aaca053e-d726-46af-a659-2a15071343c7" (UID: "aaca053e-d726-46af-a659-2a15071343c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.862020 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.861812 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istio-token" pod "e304caf1-38c3-4cbb-9c2f-db257e76ada7" (UID: "e304caf1-38c3-4cbb-9c2f-db257e76ada7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:15.862471 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.862162 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-workload-certs" pod "e304caf1-38c3-4cbb-9c2f-db257e76ada7" (UID: "e304caf1-38c3-4cbb-9c2f-db257e76ada7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.863169 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.863138 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-workload-socket" pod "aaca053e-d726-46af-a659-2a15071343c7" (UID: "aaca053e-d726-46af-a659-2a15071343c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.863425 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.863396 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istio-data" pod "e304caf1-38c3-4cbb-9c2f-db257e76ada7" (UID: "e304caf1-38c3-4cbb-9c2f-db257e76ada7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.863535 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.863458 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-istio-envoy" pod "aaca053e-d726-46af-a659-2a15071343c7" (UID: "aaca053e-d726-46af-a659-2a15071343c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.863535 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.863478 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaca053e-d726-46af-a659-2a15071343c7-istio-token" pod "aaca053e-d726-46af-a659-2a15071343c7" (UID: "aaca053e-d726-46af-a659-2a15071343c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:15.863684 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.863661 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-workload-socket" pod "e304caf1-38c3-4cbb-9c2f-db257e76ada7" (UID: "e304caf1-38c3-4cbb-9c2f-db257e76ada7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.863748 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.863712 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-workload-certs" pod "aaca053e-d726-46af-a659-2a15071343c7" (UID: "aaca053e-d726-46af-a659-2a15071343c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.864031 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.864004 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istiod-ca-cert" pod "e304caf1-38c3-4cbb-9c2f-db257e76ada7" (UID: "e304caf1-38c3-4cbb-9c2f-db257e76ada7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:15.864524 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.864499 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaca053e-d726-46af-a659-2a15071343c7-kube-api-access-6rgpz" pod "aaca053e-d726-46af-a659-2a15071343c7" (UID: "aaca053e-d726-46af-a659-2a15071343c7"). InnerVolumeSpecName "kube-api-access-6rgpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:15.865028 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.865000 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istio-envoy" pod "e304caf1-38c3-4cbb-9c2f-db257e76ada7" (UID: "e304caf1-38c3-4cbb-9c2f-db257e76ada7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.865097 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.865042 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e304caf1-38c3-4cbb-9c2f-db257e76ada7-kube-api-access-l458b" pod "e304caf1-38c3-4cbb-9c2f-db257e76ada7" (UID: "e304caf1-38c3-4cbb-9c2f-db257e76ada7"). InnerVolumeSpecName "kube-api-access-l458b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:15.865656 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.865631 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aaca053e-d726-46af-a659-2a15071343c7-istio-podinfo" pod "aaca053e-d726-46af-a659-2a15071343c7" (UID: "aaca053e-d726-46af-a659-2a15071343c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:10:15.960593 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.960558 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaca053e-d726-46af-a659-2a15071343c7-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.960593 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.960592 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.960862 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.960605 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.960862 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.960619 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.960862 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.960631 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.960862 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.960645 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l458b\" (UniqueName: \"kubernetes.io/projected/e304caf1-38c3-4cbb-9c2f-db257e76ada7-kube-api-access-l458b\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.960862 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.960658 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.960862 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.960670 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.960862 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.960683 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.960862 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.960697 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaca053e-d726-46af-a659-2a15071343c7-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.960862 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.960709 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.960862 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.960723 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.960862 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.960735 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaca053e-d726-46af-a659-2a15071343c7-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.960862 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.960747 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaca053e-d726-46af-a659-2a15071343c7-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.960862 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.960758 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.960862 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.960770 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e304caf1-38c3-4cbb-9c2f-db257e76ada7-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.960862 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.960781 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rgpz\" (UniqueName: \"kubernetes.io/projected/aaca053e-d726-46af-a659-2a15071343c7-kube-api-access-6rgpz\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.960862 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:15.960792 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e304caf1-38c3-4cbb-9c2f-db257e76ada7-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:16.045794 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.045768 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:10:16.170539 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.170473 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-credential-socket\") pod \"6183d0fa-f089-47f0-97a0-163160fc9f25\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " Nov 02 18:10:16.171094 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.171031 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-workload-socket\") pod \"6183d0fa-f089-47f0-97a0-163160fc9f25\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " Nov 02 18:10:16.171476 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.171400 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6183d0fa-f089-47f0-97a0-163160fc9f25-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6183d0fa-f089-47f0-97a0-163160fc9f25-istiod-ca-cert\") pod \"6183d0fa-f089-47f0-97a0-163160fc9f25\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " Nov 02 18:10:16.171476 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.171447 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-istio-data\") pod \"6183d0fa-f089-47f0-97a0-163160fc9f25\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " Nov 02 18:10:16.171865 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.171836 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6183d0fa-f089-47f0-97a0-163160fc9f25-istio-token\" (UniqueName: \"kubernetes.io/projected/6183d0fa-f089-47f0-97a0-163160fc9f25-istio-token\") pod \"6183d0fa-f089-47f0-97a0-163160fc9f25\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " Nov 02 18:10:16.175857 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.173634 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6183d0fa-f089-47f0-97a0-163160fc9f25-kube-api-access-pvpw2\" (UniqueName: \"kubernetes.io/projected/6183d0fa-f089-47f0-97a0-163160fc9f25-kube-api-access-pvpw2\") pod \"6183d0fa-f089-47f0-97a0-163160fc9f25\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " Nov 02 18:10:16.175857 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.173739 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-istio-envoy\") pod \"6183d0fa-f089-47f0-97a0-163160fc9f25\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " Nov 02 18:10:16.175857 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.174010 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-workload-certs\") pod \"6183d0fa-f089-47f0-97a0-163160fc9f25\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " Nov 02 18:10:16.175857 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.174128 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6183d0fa-f089-47f0-97a0-163160fc9f25-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6183d0fa-f089-47f0-97a0-163160fc9f25-istio-podinfo\") pod \"6183d0fa-f089-47f0-97a0-163160fc9f25\" (UID: \"6183d0fa-f089-47f0-97a0-163160fc9f25\") " Nov 02 18:10:16.179857 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.175994 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-istio-data" pod "6183d0fa-f089-47f0-97a0-163160fc9f25" (UID: "6183d0fa-f089-47f0-97a0-163160fc9f25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:16.179857 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.176180 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-credential-socket" pod "6183d0fa-f089-47f0-97a0-163160fc9f25" (UID: "6183d0fa-f089-47f0-97a0-163160fc9f25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:16.179857 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.176365 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-workload-socket" pod "6183d0fa-f089-47f0-97a0-163160fc9f25" (UID: "6183d0fa-f089-47f0-97a0-163160fc9f25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:16.179857 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.176628 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6183d0fa-f089-47f0-97a0-163160fc9f25-istiod-ca-cert" pod "6183d0fa-f089-47f0-97a0-163160fc9f25" (UID: "6183d0fa-f089-47f0-97a0-163160fc9f25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:16.179857 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.177472 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-workload-certs" pod "6183d0fa-f089-47f0-97a0-163160fc9f25" (UID: "6183d0fa-f089-47f0-97a0-163160fc9f25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:16.189935 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.187501 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6183d0fa-f089-47f0-97a0-163160fc9f25-istio-token" pod "6183d0fa-f089-47f0-97a0-163160fc9f25" (UID: "6183d0fa-f089-47f0-97a0-163160fc9f25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:16.190418 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.190368 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-istio-envoy" pod "6183d0fa-f089-47f0-97a0-163160fc9f25" (UID: "6183d0fa-f089-47f0-97a0-163160fc9f25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:16.192315 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.192262 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6183d0fa-f089-47f0-97a0-163160fc9f25-istio-podinfo" pod "6183d0fa-f089-47f0-97a0-163160fc9f25" (UID: "6183d0fa-f089-47f0-97a0-163160fc9f25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:10:16.193610 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.193578 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6183d0fa-f089-47f0-97a0-163160fc9f25-kube-api-access-pvpw2" pod "6183d0fa-f089-47f0-97a0-163160fc9f25" (UID: "6183d0fa-f089-47f0-97a0-163160fc9f25"). InnerVolumeSpecName "kube-api-access-pvpw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:16.200412 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.198531 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e5cafc4-6nhjm"] Nov 02 18:10:16.207591 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.207527 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4221592b-cktkl"] Nov 02 18:10:16.223757 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.215808 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e5cafc4-6nhjm"] Nov 02 18:10:16.223757 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.216227 4471 generic.go:356] "Generic (PLEG): container finished" podID="790f4f11-1fb0-49ae-871d-07295b20a266" containerID="b9e5afb1a653ef91c046b75fb888f50e24ac03adff234fed4762cbaab3baaf7f" exitCode=0 Nov 02 18:10:16.223757 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.216393 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" event={"ID":"790f4f11-1fb0-49ae-871d-07295b20a266","Type":"ContainerDied","Data":"b9e5afb1a653ef91c046b75fb888f50e24ac03adff234fed4762cbaab3baaf7f"} Nov 02 18:10:16.223757 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.216440 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" event={"ID":"790f4f11-1fb0-49ae-871d-07295b20a266","Type":"ContainerDied","Data":"4b73a36db6ac3328a497cacce764a8e6a7dcd9f3272e02b4e80004a28cb563b8"} Nov 02 18:10:16.223757 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.216470 4471 scope.go:122] "RemoveContainer" containerID="b9e5afb1a653ef91c046b75fb888f50e24ac03adff234fed4762cbaab3baaf7f" Nov 02 18:10:16.223757 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.216621 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4" Nov 02 18:10:16.223757 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.223148 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4221592b-cktkl"] Nov 02 18:10:16.227595 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.227438 4471 generic.go:356] "Generic (PLEG): container finished" podID="0b34a701-da68-40d6-9bd1-bcdd3a3e2866" containerID="1a022d508a380be3b9103abe792e805d2c3f24a8ee8ba1689b731cad78cfb6fa" exitCode=0 Nov 02 18:10:16.227595 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.227531 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5b905-fvz8z" event={"ID":"0b34a701-da68-40d6-9bd1-bcdd3a3e2866","Type":"ContainerDied","Data":"1a022d508a380be3b9103abe792e805d2c3f24a8ee8ba1689b731cad78cfb6fa"} Nov 02 18:10:16.231079 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.229396 4471 generic.go:356] "Generic (PLEG): container finished" podID="e304caf1-38c3-4cbb-9c2f-db257e76ada7" containerID="08ad3857b2a6d05b15a41996225a0a4807f64a50355e002fff80318bcf031a0f" exitCode=0 Nov 02 18:10:16.231079 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.229440 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" Nov 02 18:10:16.231079 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.229565 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" event={"ID":"e304caf1-38c3-4cbb-9c2f-db257e76ada7","Type":"ContainerDied","Data":"08ad3857b2a6d05b15a41996225a0a4807f64a50355e002fff80318bcf031a0f"} Nov 02 18:10:16.231079 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.229611 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg" event={"ID":"e304caf1-38c3-4cbb-9c2f-db257e76ada7","Type":"ContainerDied","Data":"aed1fea339a4ca98e72f7297df6396a73e62ceaa5b6da9a0b8a4aff75e21b2ef"} Nov 02 18:10:16.236215 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.234830 4471 generic.go:356] "Generic (PLEG): container finished" podID="6183d0fa-f089-47f0-97a0-163160fc9f25" containerID="033e549a2214d5dc6f831beaf5d2f08b85a6941e8511c9022f60186592b9896a" exitCode=0 Nov 02 18:10:16.236215 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.234951 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" Nov 02 18:10:16.236215 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.235029 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" event={"ID":"6183d0fa-f089-47f0-97a0-163160fc9f25","Type":"ContainerDied","Data":"033e549a2214d5dc6f831beaf5d2f08b85a6941e8511c9022f60186592b9896a"} Nov 02 18:10:16.236215 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.235072 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" event={"ID":"6183d0fa-f089-47f0-97a0-163160fc9f25","Type":"ContainerDied","Data":"193f992fdfc9ad6a3ff527b4b557f8dd88b43bdbe649e26c4dbba885d1a1cace"} Nov 02 18:10:16.245861 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.245835 4471 generic.go:356] "Generic (PLEG): container finished" podID="c1e9cee8-e915-4a1f-819d-226dcda7485a" containerID="18c0e4be2b5df625b39ee8e8cb2be8eb15ea519a0ee6d2ff9ff91e8f0c2dd0d8" exitCode=0 Nov 02 18:10:16.246034 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.246011 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" event={"ID":"c1e9cee8-e915-4a1f-819d-226dcda7485a","Type":"ContainerDied","Data":"18c0e4be2b5df625b39ee8e8cb2be8eb15ea519a0ee6d2ff9ff91e8f0c2dd0d8"} Nov 02 18:10:16.246114 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.246061 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" event={"ID":"c1e9cee8-e915-4a1f-819d-226dcda7485a","Type":"ContainerDied","Data":"42a6e06382c9794653f26594cb26be3397af216fe0c06df593264c8e28901888"} Nov 02 18:10:16.246172 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.246139 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" Nov 02 18:10:16.248122 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.247968 4471 generic.go:356] "Generic (PLEG): container finished" podID="aaca053e-d726-46af-a659-2a15071343c7" containerID="d0b8e1501153337b07e1139059c008fa0659d99849d80e336abecb53916df430" exitCode=0 Nov 02 18:10:16.248122 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.248027 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" event={"ID":"aaca053e-d726-46af-a659-2a15071343c7","Type":"ContainerDied","Data":"d0b8e1501153337b07e1139059c008fa0659d99849d80e336abecb53916df430"} Nov 02 18:10:16.248122 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.248063 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" event={"ID":"aaca053e-d726-46af-a659-2a15071343c7","Type":"ContainerDied","Data":"737fc06c97e32d47f4f48b2b15ea6bd3c10d4160b079d95343fa616768f472fa"} Nov 02 18:10:16.248789 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.248767 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c" Nov 02 18:10:16.263715 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.263515 4471 scope.go:122] "RemoveContainer" containerID="b9e5afb1a653ef91c046b75fb888f50e24ac03adff234fed4762cbaab3baaf7f" Nov 02 18:10:16.263929 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:10:16.263908 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9e5afb1a653ef91c046b75fb888f50e24ac03adff234fed4762cbaab3baaf7f\": container with ID starting with b9e5afb1a653ef91c046b75fb888f50e24ac03adff234fed4762cbaab3baaf7f not found: ID does not exist" containerID="b9e5afb1a653ef91c046b75fb888f50e24ac03adff234fed4762cbaab3baaf7f" Nov 02 18:10:16.264018 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.263942 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9e5afb1a653ef91c046b75fb888f50e24ac03adff234fed4762cbaab3baaf7f"} err="failed to get container status \"b9e5afb1a653ef91c046b75fb888f50e24ac03adff234fed4762cbaab3baaf7f\": rpc error: code = NotFound desc = could not find container \"b9e5afb1a653ef91c046b75fb888f50e24ac03adff234fed4762cbaab3baaf7f\": container with ID starting with b9e5afb1a653ef91c046b75fb888f50e24ac03adff234fed4762cbaab3baaf7f not found: ID does not exist" Nov 02 18:10:16.264018 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.263966 4471 scope.go:122] "RemoveContainer" containerID="08ad3857b2a6d05b15a41996225a0a4807f64a50355e002fff80318bcf031a0f" Nov 02 18:10:16.276132 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.276109 4471 scope.go:122] "RemoveContainer" containerID="08ad3857b2a6d05b15a41996225a0a4807f64a50355e002fff80318bcf031a0f" Nov 02 18:10:16.276499 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:10:16.276471 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08ad3857b2a6d05b15a41996225a0a4807f64a50355e002fff80318bcf031a0f\": container with ID starting with 08ad3857b2a6d05b15a41996225a0a4807f64a50355e002fff80318bcf031a0f not found: ID does not exist" containerID="08ad3857b2a6d05b15a41996225a0a4807f64a50355e002fff80318bcf031a0f" Nov 02 18:10:16.276621 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.276505 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08ad3857b2a6d05b15a41996225a0a4807f64a50355e002fff80318bcf031a0f"} err="failed to get container status \"08ad3857b2a6d05b15a41996225a0a4807f64a50355e002fff80318bcf031a0f\": rpc error: code = NotFound desc = could not find container \"08ad3857b2a6d05b15a41996225a0a4807f64a50355e002fff80318bcf031a0f\": container with ID starting with 08ad3857b2a6d05b15a41996225a0a4807f64a50355e002fff80318bcf031a0f not found: ID does not exist" Nov 02 18:10:16.276621 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.276525 4471 scope.go:122] "RemoveContainer" containerID="033e549a2214d5dc6f831beaf5d2f08b85a6941e8511c9022f60186592b9896a" Nov 02 18:10:16.277056 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.277024 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:16.277056 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.277052 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:16.277227 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.277062 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6183d0fa-f089-47f0-97a0-163160fc9f25-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:16.277227 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.277071 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:16.277227 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.277079 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6183d0fa-f089-47f0-97a0-163160fc9f25-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:16.277227 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.277091 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvpw2\" (UniqueName: \"kubernetes.io/projected/6183d0fa-f089-47f0-97a0-163160fc9f25-kube-api-access-pvpw2\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:16.277227 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.277107 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:16.277227 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.277120 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6183d0fa-f089-47f0-97a0-163160fc9f25-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:16.277227 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.277132 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6183d0fa-f089-47f0-97a0-163160fc9f25-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:16.293796 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.293768 4471 scope.go:122] "RemoveContainer" containerID="033e549a2214d5dc6f831beaf5d2f08b85a6941e8511c9022f60186592b9896a" Nov 02 18:10:16.294131 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:10:16.294107 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"033e549a2214d5dc6f831beaf5d2f08b85a6941e8511c9022f60186592b9896a\": container with ID starting with 033e549a2214d5dc6f831beaf5d2f08b85a6941e8511c9022f60186592b9896a not found: ID does not exist" containerID="033e549a2214d5dc6f831beaf5d2f08b85a6941e8511c9022f60186592b9896a" Nov 02 18:10:16.294182 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.294142 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"033e549a2214d5dc6f831beaf5d2f08b85a6941e8511c9022f60186592b9896a"} err="failed to get container status \"033e549a2214d5dc6f831beaf5d2f08b85a6941e8511c9022f60186592b9896a\": rpc error: code = NotFound desc = could not find container \"033e549a2214d5dc6f831beaf5d2f08b85a6941e8511c9022f60186592b9896a\": container with ID starting with 033e549a2214d5dc6f831beaf5d2f08b85a6941e8511c9022f60186592b9896a not found: ID does not exist" Nov 02 18:10:16.294182 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.294164 4471 scope.go:122] "RemoveContainer" containerID="18c0e4be2b5df625b39ee8e8cb2be8eb15ea519a0ee6d2ff9ff91e8f0c2dd0d8" Nov 02 18:10:16.304363 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.304345 4471 scope.go:122] "RemoveContainer" containerID="18c0e4be2b5df625b39ee8e8cb2be8eb15ea519a0ee6d2ff9ff91e8f0c2dd0d8" Nov 02 18:10:16.304613 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:10:16.304596 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18c0e4be2b5df625b39ee8e8cb2be8eb15ea519a0ee6d2ff9ff91e8f0c2dd0d8\": container with ID starting with 18c0e4be2b5df625b39ee8e8cb2be8eb15ea519a0ee6d2ff9ff91e8f0c2dd0d8 not found: ID does not exist" containerID="18c0e4be2b5df625b39ee8e8cb2be8eb15ea519a0ee6d2ff9ff91e8f0c2dd0d8" Nov 02 18:10:16.304682 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.304618 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18c0e4be2b5df625b39ee8e8cb2be8eb15ea519a0ee6d2ff9ff91e8f0c2dd0d8"} err="failed to get container status \"18c0e4be2b5df625b39ee8e8cb2be8eb15ea519a0ee6d2ff9ff91e8f0c2dd0d8\": rpc error: code = NotFound desc = could not find container \"18c0e4be2b5df625b39ee8e8cb2be8eb15ea519a0ee6d2ff9ff91e8f0c2dd0d8\": container with ID starting with 18c0e4be2b5df625b39ee8e8cb2be8eb15ea519a0ee6d2ff9ff91e8f0c2dd0d8 not found: ID does not exist" Nov 02 18:10:16.304682 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.304634 4471 scope.go:122] "RemoveContainer" containerID="d0b8e1501153337b07e1139059c008fa0659d99849d80e336abecb53916df430" Nov 02 18:10:16.314894 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.314867 4471 scope.go:122] "RemoveContainer" containerID="d0b8e1501153337b07e1139059c008fa0659d99849d80e336abecb53916df430" Nov 02 18:10:16.315120 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:10:16.315100 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0b8e1501153337b07e1139059c008fa0659d99849d80e336abecb53916df430\": container with ID starting with d0b8e1501153337b07e1139059c008fa0659d99849d80e336abecb53916df430 not found: ID does not exist" containerID="d0b8e1501153337b07e1139059c008fa0659d99849d80e336abecb53916df430" Nov 02 18:10:16.315210 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.315126 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0b8e1501153337b07e1139059c008fa0659d99849d80e336abecb53916df430"} err="failed to get container status \"d0b8e1501153337b07e1139059c008fa0659d99849d80e336abecb53916df430\": rpc error: code = NotFound desc = could not find container \"d0b8e1501153337b07e1139059c008fa0659d99849d80e336abecb53916df430\": container with ID starting with d0b8e1501153337b07e1139059c008fa0659d99849d80e336abecb53916df430 not found: ID does not exist" Nov 02 18:10:16.316003 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.315978 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rt5cp_1b5ee10f-87a1-4168-904e-ba233f304fe3/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:16.369276 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.369245 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597"] Nov 02 18:10:16.384113 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.384069 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597"] Nov 02 18:10:16.407489 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.407460 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl"] Nov 02 18:10:16.414218 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.414188 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl"] Nov 02 18:10:16.444396 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.444358 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="517d2554-a194-476c-a192-d60a6675e38a" path="/var/lib/kubelet/pods/517d2554-a194-476c-a192-d60a6675e38a/volumes" Nov 02 18:10:16.445478 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.445168 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6183d0fa-f089-47f0-97a0-163160fc9f25" path="/var/lib/kubelet/pods/6183d0fa-f089-47f0-97a0-163160fc9f25/volumes" Nov 02 18:10:16.445812 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.445792 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1e9cee8-e915-4a1f-819d-226dcda7485a" path="/var/lib/kubelet/pods/c1e9cee8-e915-4a1f-819d-226dcda7485a/volumes" Nov 02 18:10:16.446642 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.446625 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf19245a-0076-43fe-934f-e0f8f3ac19c4" path="/var/lib/kubelet/pods/cf19245a-0076-43fe-934f-e0f8f3ac19c4/volumes" Nov 02 18:10:16.450169 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.450145 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4"] Nov 02 18:10:16.458629 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.458576 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-rf5j4"] Nov 02 18:10:16.496683 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.496648 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c"] Nov 02 18:10:16.503041 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.503013 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-9fv4c"] Nov 02 18:10:16.523814 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.523788 4471 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:10:16.523934 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.523850 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-bt597" podUID="c1e9cee8-e915-4a1f-819d-226dcda7485a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:10:16.535643 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.535617 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg"] Nov 02 18:10:16.539836 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.539816 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-qrgrg"] Nov 02 18:10:16.956781 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.956741 4471 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:10:16.957227 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:16.956801 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-zw5jl" podUID="6183d0fa-f089-47f0-97a0-163160fc9f25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:10:17.441893 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:17.441868 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5b905-fvz8z" Nov 02 18:10:17.604769 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:17.604683 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b34a701-da68-40d6-9bd1-bcdd3a3e2866-kube-api-access-5flc7\" (UniqueName: \"kubernetes.io/projected/0b34a701-da68-40d6-9bd1-bcdd3a3e2866-kube-api-access-5flc7\") pod \"0b34a701-da68-40d6-9bd1-bcdd3a3e2866\" (UID: \"0b34a701-da68-40d6-9bd1-bcdd3a3e2866\") " Nov 02 18:10:17.604964 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:17.604943 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b34a701-da68-40d6-9bd1-bcdd3a3e2866-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b34a701-da68-40d6-9bd1-bcdd3a3e2866-maas-api-xks-ca\") pod \"0b34a701-da68-40d6-9bd1-bcdd3a3e2866\" (UID: \"0b34a701-da68-40d6-9bd1-bcdd3a3e2866\") " Nov 02 18:10:17.605106 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:17.605090 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b34a701-da68-40d6-9bd1-bcdd3a3e2866-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b34a701-da68-40d6-9bd1-bcdd3a3e2866-maas-api-openshift-ca\") pod \"0b34a701-da68-40d6-9bd1-bcdd3a3e2866\" (UID: \"0b34a701-da68-40d6-9bd1-bcdd3a3e2866\") " Nov 02 18:10:17.605500 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:17.605470 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b34a701-da68-40d6-9bd1-bcdd3a3e2866-maas-api-openshift-ca" pod "0b34a701-da68-40d6-9bd1-bcdd3a3e2866" (UID: "0b34a701-da68-40d6-9bd1-bcdd3a3e2866"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:17.605876 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:17.605859 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b34a701-da68-40d6-9bd1-bcdd3a3e2866-maas-api-openshift-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:17.607020 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:17.606998 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b34a701-da68-40d6-9bd1-bcdd3a3e2866-maas-api-xks-ca" pod "0b34a701-da68-40d6-9bd1-bcdd3a3e2866" (UID: "0b34a701-da68-40d6-9bd1-bcdd3a3e2866"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:10:17.607129 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:17.607117 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b34a701-da68-40d6-9bd1-bcdd3a3e2866-kube-api-access-5flc7" pod "0b34a701-da68-40d6-9bd1-bcdd3a3e2866" (UID: "0b34a701-da68-40d6-9bd1-bcdd3a3e2866"). InnerVolumeSpecName "kube-api-access-5flc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:17.707342 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:17.707297 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5flc7\" (UniqueName: \"kubernetes.io/projected/0b34a701-da68-40d6-9bd1-bcdd3a3e2866-kube-api-access-5flc7\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:17.707342 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:17.707339 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b34a701-da68-40d6-9bd1-bcdd3a3e2866-maas-api-xks-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:18.277397 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.276489 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5b905-fvz8z" Nov 02 18:10:18.277397 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.276519 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5b905-fvz8z" event={"ID":"0b34a701-da68-40d6-9bd1-bcdd3a3e2866","Type":"ContainerDied","Data":"96e6f5917a205fdbae30e0beb8f915bbd0bb3450ecf00c9fc5a38df76f1d903b"} Nov 02 18:10:18.277397 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.276572 4471 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96e6f5917a205fdbae30e0beb8f915bbd0bb3450ecf00c9fc5a38df76f1d903b" Nov 02 18:10:18.282867 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.282588 4471 generic.go:356] "Generic (PLEG): container finished" podID="19632b35-0e41-4d90-89c3-ced789733fbc" containerID="f2aae250a184d3f9120231114c61a5f8aceebfd1e61d53603be12066a16e1b01" exitCode=0 Nov 02 18:10:18.282867 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.282708 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" event={"ID":"19632b35-0e41-4d90-89c3-ced789733fbc","Type":"ContainerDied","Data":"f2aae250a184d3f9120231114c61a5f8aceebfd1e61d53603be12066a16e1b01"} Nov 02 18:10:18.282867 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.282760 4471 scope.go:122] "RemoveContainer" containerID="f9796c87e49aaa1a9c983d5d0155d4d5208bc74f1f4deef29a67a86286af22b7" Nov 02 18:10:18.429437 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.429409 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 18:10:18.443017 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.442972 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="790f4f11-1fb0-49ae-871d-07295b20a266" path="/var/lib/kubelet/pods/790f4f11-1fb0-49ae-871d-07295b20a266/volumes" Nov 02 18:10:18.443780 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.443759 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaca053e-d726-46af-a659-2a15071343c7" path="/var/lib/kubelet/pods/aaca053e-d726-46af-a659-2a15071343c7/volumes" Nov 02 18:10:18.444463 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.444443 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e304caf1-38c3-4cbb-9c2f-db257e76ada7" path="/var/lib/kubelet/pods/e304caf1-38c3-4cbb-9c2f-db257e76ada7/volumes" Nov 02 18:10:18.523359 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.523323 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19632b35-0e41-4d90-89c3-ced789733fbc-istio-token\" (UniqueName: \"kubernetes.io/projected/19632b35-0e41-4d90-89c3-ced789733fbc-istio-token\") pod \"19632b35-0e41-4d90-89c3-ced789733fbc\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " Nov 02 18:10:18.523553 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.523433 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-workload-certs\") pod \"19632b35-0e41-4d90-89c3-ced789733fbc\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " Nov 02 18:10:18.523553 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.523502 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-credential-socket\") pod \"19632b35-0e41-4d90-89c3-ced789733fbc\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " Nov 02 18:10:18.523952 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.523638 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19632b35-0e41-4d90-89c3-ced789733fbc-kube-api-access-vwhbk\" (UniqueName: \"kubernetes.io/projected/19632b35-0e41-4d90-89c3-ced789733fbc-kube-api-access-vwhbk\") pod \"19632b35-0e41-4d90-89c3-ced789733fbc\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " Nov 02 18:10:18.523952 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.523752 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19632b35-0e41-4d90-89c3-ced789733fbc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19632b35-0e41-4d90-89c3-ced789733fbc-istio-podinfo\") pod \"19632b35-0e41-4d90-89c3-ced789733fbc\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " Nov 02 18:10:18.523952 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.523836 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-workload-certs" pod "19632b35-0e41-4d90-89c3-ced789733fbc" (UID: "19632b35-0e41-4d90-89c3-ced789733fbc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:18.523952 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.523856 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-credential-socket" pod "19632b35-0e41-4d90-89c3-ced789733fbc" (UID: "19632b35-0e41-4d90-89c3-ced789733fbc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:18.523952 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.523912 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-workload-socket\") pod \"19632b35-0e41-4d90-89c3-ced789733fbc\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " Nov 02 18:10:18.524449 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.524247 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-workload-socket" pod "19632b35-0e41-4d90-89c3-ced789733fbc" (UID: "19632b35-0e41-4d90-89c3-ced789733fbc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:18.524449 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.523950 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-istio-data\") pod \"19632b35-0e41-4d90-89c3-ced789733fbc\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " Nov 02 18:10:18.524449 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.524379 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-istio-data" pod "19632b35-0e41-4d90-89c3-ced789733fbc" (UID: "19632b35-0e41-4d90-89c3-ced789733fbc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:18.524449 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.524435 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-istio-envoy\") pod \"19632b35-0e41-4d90-89c3-ced789733fbc\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " Nov 02 18:10:18.524674 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.524468 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19632b35-0e41-4d90-89c3-ced789733fbc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19632b35-0e41-4d90-89c3-ced789733fbc-istiod-ca-cert\") pod \"19632b35-0e41-4d90-89c3-ced789733fbc\" (UID: \"19632b35-0e41-4d90-89c3-ced789733fbc\") " Nov 02 18:10:18.525283 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.525255 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19632b35-0e41-4d90-89c3-ced789733fbc-istiod-ca-cert" pod "19632b35-0e41-4d90-89c3-ced789733fbc" (UID: "19632b35-0e41-4d90-89c3-ced789733fbc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:18.525466 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.525441 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:18.525466 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.525467 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:18.525661 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.525482 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19632b35-0e41-4d90-89c3-ced789733fbc-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:18.525661 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.525495 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:18.525661 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.525508 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:18.526416 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.526383 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19632b35-0e41-4d90-89c3-ced789733fbc-istio-token" pod "19632b35-0e41-4d90-89c3-ced789733fbc" (UID: "19632b35-0e41-4d90-89c3-ced789733fbc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:18.526641 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.526622 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19632b35-0e41-4d90-89c3-ced789733fbc-istio-podinfo" pod "19632b35-0e41-4d90-89c3-ced789733fbc" (UID: "19632b35-0e41-4d90-89c3-ced789733fbc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:10:18.527092 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.527070 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-istio-envoy" pod "19632b35-0e41-4d90-89c3-ced789733fbc" (UID: "19632b35-0e41-4d90-89c3-ced789733fbc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:18.527145 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.527111 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19632b35-0e41-4d90-89c3-ced789733fbc-kube-api-access-vwhbk" pod "19632b35-0e41-4d90-89c3-ced789733fbc" (UID: "19632b35-0e41-4d90-89c3-ced789733fbc"). InnerVolumeSpecName "kube-api-access-vwhbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:18.627052 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.627015 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwhbk\" (UniqueName: \"kubernetes.io/projected/19632b35-0e41-4d90-89c3-ced789733fbc-kube-api-access-vwhbk\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:18.627052 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.627046 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19632b35-0e41-4d90-89c3-ced789733fbc-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:18.627052 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.627059 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19632b35-0e41-4d90-89c3-ced789733fbc-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:18.627345 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:18.627070 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19632b35-0e41-4d90-89c3-ced789733fbc-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:19.292116 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:19.292083 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" Nov 02 18:10:19.292613 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:19.292114 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs" event={"ID":"19632b35-0e41-4d90-89c3-ced789733fbc","Type":"ContainerDied","Data":"eefc2e420fee287b8e71cc04e1ea19117660d07bf99a7eada1eb077dfe4583ab"} Nov 02 18:10:19.292613 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:19.292171 4471 scope.go:122] "RemoveContainer" containerID="f2aae250a184d3f9120231114c61a5f8aceebfd1e61d53603be12066a16e1b01" Nov 02 18:10:19.360975 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:19.360942 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs"] Nov 02 18:10:19.373900 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:19.373864 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-bb6d2e-openshift-default-7d5f79cd49-hp8cs"] Nov 02 18:10:20.445280 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:20.445243 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19632b35-0e41-4d90-89c3-ced789733fbc" path="/var/lib/kubelet/pods/19632b35-0e41-4d90-89c3-ced789733fbc/volumes" Nov 02 18:10:20.852389 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:20.852355 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b2fc9f-6598c4cdc5-f5rrt"] Nov 02 18:10:20.891475 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:20.891438 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b2fc9f-5f789b6cdc-sdwdq"] Nov 02 18:10:21.779367 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:21.778785 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5fd3e0-74cfc7bbf7-cc5bk"] Nov 02 18:10:21.869373 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:21.866949 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5fd3e0-77f74bdccf-xd8fb"] Nov 02 18:10:24.296426 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:24.296391 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d5b905-6bc9bc867-zg4xp"] Nov 02 18:10:25.852663 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:25.852623 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-b2fc9f-6598c4cdc5-f5rrt" podUID="2bc68f8a-08a7-49b2-9564-f8041c6d96e0" containerName="payload-processing" containerID="cri-o://4765540b783a51ac53326f08362de9a6f6cb511211083d2cdb0204211e7b5eae" gracePeriod=25 Nov 02 18:10:25.892078 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:25.892013 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2fc9f-5f789b6cdc-sdwdq" podUID="03f6f5cc-5479-478e-8a95-b86c359d2935" containerName="payload-pre-processing" containerID="cri-o://3ea52fa1275e9053a773e2c2832774fad4d159f1b04614ff468d6013840a6f60" gracePeriod=25 Nov 02 18:10:26.199300 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.199263 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b2fc9f-6598c4cdc5-f5rrt" Nov 02 18:10:26.230076 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.230051 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2fc9f-5f789b6cdc-sdwdq" Nov 02 18:10:26.301512 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.301473 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03f6f5cc-5479-478e-8a95-b86c359d2935-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03f6f5cc-5479-478e-8a95-b86c359d2935-plugins-config-volume\") pod \"03f6f5cc-5479-478e-8a95-b86c359d2935\" (UID: \"03f6f5cc-5479-478e-8a95-b86c359d2935\") " Nov 02 18:10:26.301776 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.301689 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03f6f5cc-5479-478e-8a95-b86c359d2935-kube-api-access-jcfn9\" (UniqueName: \"kubernetes.io/projected/03f6f5cc-5479-478e-8a95-b86c359d2935-kube-api-access-jcfn9\") pod \"03f6f5cc-5479-478e-8a95-b86c359d2935\" (UID: \"03f6f5cc-5479-478e-8a95-b86c359d2935\") " Nov 02 18:10:26.301915 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.301898 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bc68f8a-08a7-49b2-9564-f8041c6d96e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bc68f8a-08a7-49b2-9564-f8041c6d96e0-plugins-config-volume\") pod \"2bc68f8a-08a7-49b2-9564-f8041c6d96e0\" (UID: \"2bc68f8a-08a7-49b2-9564-f8041c6d96e0\") " Nov 02 18:10:26.301980 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.301919 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03f6f5cc-5479-478e-8a95-b86c359d2935-plugins-config-volume" pod "03f6f5cc-5479-478e-8a95-b86c359d2935" (UID: "03f6f5cc-5479-478e-8a95-b86c359d2935"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:26.302293 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.302271 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bc68f8a-08a7-49b2-9564-f8041c6d96e0-plugins-config-volume" pod "2bc68f8a-08a7-49b2-9564-f8041c6d96e0" (UID: "2bc68f8a-08a7-49b2-9564-f8041c6d96e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:26.302407 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.302379 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bc68f8a-08a7-49b2-9564-f8041c6d96e0-kube-api-access-vzlgd\" (UniqueName: \"kubernetes.io/projected/2bc68f8a-08a7-49b2-9564-f8041c6d96e0-kube-api-access-vzlgd\") pod \"2bc68f8a-08a7-49b2-9564-f8041c6d96e0\" (UID: \"2bc68f8a-08a7-49b2-9564-f8041c6d96e0\") " Nov 02 18:10:26.303418 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.303393 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bc68f8a-08a7-49b2-9564-f8041c6d96e0-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:26.303520 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.303422 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03f6f5cc-5479-478e-8a95-b86c359d2935-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:26.304390 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.304362 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f6f5cc-5479-478e-8a95-b86c359d2935-kube-api-access-jcfn9" pod "03f6f5cc-5479-478e-8a95-b86c359d2935" (UID: "03f6f5cc-5479-478e-8a95-b86c359d2935"). InnerVolumeSpecName "kube-api-access-jcfn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:26.304670 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.304645 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bc68f8a-08a7-49b2-9564-f8041c6d96e0-kube-api-access-vzlgd" pod "2bc68f8a-08a7-49b2-9564-f8041c6d96e0" (UID: "2bc68f8a-08a7-49b2-9564-f8041c6d96e0"). InnerVolumeSpecName "kube-api-access-vzlgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:26.355142 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.355108 4471 generic.go:356] "Generic (PLEG): container finished" podID="2bc68f8a-08a7-49b2-9564-f8041c6d96e0" containerID="4765540b783a51ac53326f08362de9a6f6cb511211083d2cdb0204211e7b5eae" exitCode=0 Nov 02 18:10:26.355267 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.355151 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b2fc9f-6598c4cdc5-f5rrt" Nov 02 18:10:26.355267 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.355183 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b2fc9f-6598c4cdc5-f5rrt" event={"ID":"2bc68f8a-08a7-49b2-9564-f8041c6d96e0","Type":"ContainerDied","Data":"4765540b783a51ac53326f08362de9a6f6cb511211083d2cdb0204211e7b5eae"} Nov 02 18:10:26.355267 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.355238 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b2fc9f-6598c4cdc5-f5rrt" event={"ID":"2bc68f8a-08a7-49b2-9564-f8041c6d96e0","Type":"ContainerDied","Data":"1d39d295c0f4d645e86b09473221d634d3d89caa1d113e61a5658e49e92676b9"} Nov 02 18:10:26.355267 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.355262 4471 scope.go:122] "RemoveContainer" containerID="4765540b783a51ac53326f08362de9a6f6cb511211083d2cdb0204211e7b5eae" Nov 02 18:10:26.357387 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.357357 4471 generic.go:356] "Generic (PLEG): container finished" podID="03f6f5cc-5479-478e-8a95-b86c359d2935" containerID="3ea52fa1275e9053a773e2c2832774fad4d159f1b04614ff468d6013840a6f60" exitCode=0 Nov 02 18:10:26.357499 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.357403 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2fc9f-5f789b6cdc-sdwdq" event={"ID":"03f6f5cc-5479-478e-8a95-b86c359d2935","Type":"ContainerDied","Data":"3ea52fa1275e9053a773e2c2832774fad4d159f1b04614ff468d6013840a6f60"} Nov 02 18:10:26.357499 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.357414 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2fc9f-5f789b6cdc-sdwdq" Nov 02 18:10:26.357499 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.357448 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2fc9f-5f789b6cdc-sdwdq" event={"ID":"03f6f5cc-5479-478e-8a95-b86c359d2935","Type":"ContainerDied","Data":"92cc52e265cb7d44b0ac8028164bf92f25d6b1a5bb33acbfbb53e983e23e3bfa"} Nov 02 18:10:26.366807 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.366789 4471 scope.go:122] "RemoveContainer" containerID="4765540b783a51ac53326f08362de9a6f6cb511211083d2cdb0204211e7b5eae" Nov 02 18:10:26.367127 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:10:26.367101 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4765540b783a51ac53326f08362de9a6f6cb511211083d2cdb0204211e7b5eae\": container with ID starting with 4765540b783a51ac53326f08362de9a6f6cb511211083d2cdb0204211e7b5eae not found: ID does not exist" containerID="4765540b783a51ac53326f08362de9a6f6cb511211083d2cdb0204211e7b5eae" Nov 02 18:10:26.367236 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.367138 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4765540b783a51ac53326f08362de9a6f6cb511211083d2cdb0204211e7b5eae"} err="failed to get container status \"4765540b783a51ac53326f08362de9a6f6cb511211083d2cdb0204211e7b5eae\": rpc error: code = NotFound desc = could not find container \"4765540b783a51ac53326f08362de9a6f6cb511211083d2cdb0204211e7b5eae\": container with ID starting with 4765540b783a51ac53326f08362de9a6f6cb511211083d2cdb0204211e7b5eae not found: ID does not exist" Nov 02 18:10:26.367236 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.367162 4471 scope.go:122] "RemoveContainer" containerID="3ea52fa1275e9053a773e2c2832774fad4d159f1b04614ff468d6013840a6f60" Nov 02 18:10:26.377683 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.377665 4471 scope.go:122] "RemoveContainer" containerID="3ea52fa1275e9053a773e2c2832774fad4d159f1b04614ff468d6013840a6f60" Nov 02 18:10:26.377979 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:10:26.377960 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ea52fa1275e9053a773e2c2832774fad4d159f1b04614ff468d6013840a6f60\": container with ID starting with 3ea52fa1275e9053a773e2c2832774fad4d159f1b04614ff468d6013840a6f60 not found: ID does not exist" containerID="3ea52fa1275e9053a773e2c2832774fad4d159f1b04614ff468d6013840a6f60" Nov 02 18:10:26.378035 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.377986 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ea52fa1275e9053a773e2c2832774fad4d159f1b04614ff468d6013840a6f60"} err="failed to get container status \"3ea52fa1275e9053a773e2c2832774fad4d159f1b04614ff468d6013840a6f60\": rpc error: code = NotFound desc = could not find container \"3ea52fa1275e9053a773e2c2832774fad4d159f1b04614ff468d6013840a6f60\": container with ID starting with 3ea52fa1275e9053a773e2c2832774fad4d159f1b04614ff468d6013840a6f60 not found: ID does not exist" Nov 02 18:10:26.405248 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.405190 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcfn9\" (UniqueName: \"kubernetes.io/projected/03f6f5cc-5479-478e-8a95-b86c359d2935-kube-api-access-jcfn9\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:26.405248 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.405215 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzlgd\" (UniqueName: \"kubernetes.io/projected/2bc68f8a-08a7-49b2-9564-f8041c6d96e0-kube-api-access-vzlgd\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:26.423400 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.423365 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b2fc9f-6598c4cdc5-f5rrt"] Nov 02 18:10:26.423615 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.423601 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b2fc9f-6598c4cdc5-f5rrt"] Nov 02 18:10:26.444544 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.444513 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bc68f8a-08a7-49b2-9564-f8041c6d96e0" path="/var/lib/kubelet/pods/2bc68f8a-08a7-49b2-9564-f8041c6d96e0/volumes" Nov 02 18:10:26.451955 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.451929 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b2fc9f-5f789b6cdc-sdwdq"] Nov 02 18:10:26.457668 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.457646 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b2fc9f-5f789b6cdc-sdwdq"] Nov 02 18:10:26.779476 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.779421 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-5fd3e0-74cfc7bbf7-cc5bk" podUID="1c0b125a-116e-46b0-ad0b-0f7ff15fefa6" containerName="payload-processing" containerID="cri-o://6127dcc0d44863ba678229d696cb6036d9d70b5ce71298022129cca0cbdb47bc" gracePeriod=25 Nov 02 18:10:26.867848 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:26.867788 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5fd3e0-77f74bdccf-xd8fb" podUID="d63f9cab-764f-46ae-a522-2ef02346c21a" containerName="payload-pre-processing" containerID="cri-o://86d37e9151ad330bee33c9c89ebeeb0033c7876b4e997a9b89b3a3355eda179d" gracePeriod=25 Nov 02 18:10:27.192328 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:27.160488 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5fd3e0-74cfc7bbf7-cc5bk" Nov 02 18:10:27.254564 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:27.254529 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c0b125a-116e-46b0-ad0b-0f7ff15fefa6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0b125a-116e-46b0-ad0b-0f7ff15fefa6-plugins-config-volume\") pod \"1c0b125a-116e-46b0-ad0b-0f7ff15fefa6\" (UID: \"1c0b125a-116e-46b0-ad0b-0f7ff15fefa6\") " Nov 02 18:10:27.255356 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:27.255297 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c0b125a-116e-46b0-ad0b-0f7ff15fefa6-plugins-config-volume" pod "1c0b125a-116e-46b0-ad0b-0f7ff15fefa6" (UID: "1c0b125a-116e-46b0-ad0b-0f7ff15fefa6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:27.267756 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:27.264673 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c0b125a-116e-46b0-ad0b-0f7ff15fefa6-kube-api-access-l7wrk\" (UniqueName: \"kubernetes.io/projected/1c0b125a-116e-46b0-ad0b-0f7ff15fefa6-kube-api-access-l7wrk\") pod \"1c0b125a-116e-46b0-ad0b-0f7ff15fefa6\" (UID: \"1c0b125a-116e-46b0-ad0b-0f7ff15fefa6\") " Nov 02 18:10:27.267756 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:27.265706 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0b125a-116e-46b0-ad0b-0f7ff15fefa6-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:27.267756 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:27.267362 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c0b125a-116e-46b0-ad0b-0f7ff15fefa6-kube-api-access-l7wrk" pod "1c0b125a-116e-46b0-ad0b-0f7ff15fefa6" (UID: "1c0b125a-116e-46b0-ad0b-0f7ff15fefa6"). InnerVolumeSpecName "kube-api-access-l7wrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:27.272764 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:27.272735 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5fd3e0-77f74bdccf-xd8fb" Nov 02 18:10:27.366623 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:27.366524 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d63f9cab-764f-46ae-a522-2ef02346c21a-kube-api-access-z7pgj\" (UniqueName: \"kubernetes.io/projected/d63f9cab-764f-46ae-a522-2ef02346c21a-kube-api-access-z7pgj\") pod \"d63f9cab-764f-46ae-a522-2ef02346c21a\" (UID: \"d63f9cab-764f-46ae-a522-2ef02346c21a\") " Nov 02 18:10:27.366799 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:27.366733 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d63f9cab-764f-46ae-a522-2ef02346c21a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d63f9cab-764f-46ae-a522-2ef02346c21a-plugins-config-volume\") pod \"d63f9cab-764f-46ae-a522-2ef02346c21a\" (UID: \"d63f9cab-764f-46ae-a522-2ef02346c21a\") " Nov 02 18:10:27.367412 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:27.367367 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d63f9cab-764f-46ae-a522-2ef02346c21a-plugins-config-volume" pod "d63f9cab-764f-46ae-a522-2ef02346c21a" (UID: "d63f9cab-764f-46ae-a522-2ef02346c21a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:27.367636 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:27.367612 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7wrk\" (UniqueName: \"kubernetes.io/projected/1c0b125a-116e-46b0-ad0b-0f7ff15fefa6-kube-api-access-l7wrk\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:27.367728 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:27.367641 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d63f9cab-764f-46ae-a522-2ef02346c21a-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:27.369027 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:27.368999 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d63f9cab-764f-46ae-a522-2ef02346c21a-kube-api-access-z7pgj" pod "d63f9cab-764f-46ae-a522-2ef02346c21a" (UID: "d63f9cab-764f-46ae-a522-2ef02346c21a"). InnerVolumeSpecName "kube-api-access-z7pgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:27.374102 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:27.374073 4471 generic.go:356] "Generic (PLEG): container finished" podID="1c0b125a-116e-46b0-ad0b-0f7ff15fefa6" containerID="6127dcc0d44863ba678229d696cb6036d9d70b5ce71298022129cca0cbdb47bc" exitCode=0 Nov 02 18:10:27.374203 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:27.374144 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5fd3e0-74cfc7bbf7-cc5bk" Nov 02 18:10:27.374203 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:27.374172 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5fd3e0-74cfc7bbf7-cc5bk" event={"ID":"1c0b125a-116e-46b0-ad0b-0f7ff15fefa6","Type":"ContainerDied","Data":"6127dcc0d44863ba678229d696cb6036d9d70b5ce71298022129cca0cbdb47bc"} Nov 02 18:10:27.374385 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:27.374238 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5fd3e0-74cfc7bbf7-cc5bk" event={"ID":"1c0b125a-116e-46b0-ad0b-0f7ff15fefa6","Type":"ContainerDied","Data":"0e689da343c04125d352d43ed448689fb08caf0f0ca0fe7cb7536419438e287c"} Nov 02 18:10:27.374385 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:27.374272 4471 scope.go:122] "RemoveContainer" containerID="6127dcc0d44863ba678229d696cb6036d9d70b5ce71298022129cca0cbdb47bc" Nov 02 18:10:27.376371 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:27.376346 4471 generic.go:356] "Generic (PLEG): container finished" podID="d63f9cab-764f-46ae-a522-2ef02346c21a" containerID="86d37e9151ad330bee33c9c89ebeeb0033c7876b4e997a9b89b3a3355eda179d" exitCode=0 Nov 02 18:10:27.376489 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:27.376441 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5fd3e0-77f74bdccf-xd8fb" Nov 02 18:10:27.376593 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:27.376573 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5fd3e0-77f74bdccf-xd8fb" event={"ID":"d63f9cab-764f-46ae-a522-2ef02346c21a","Type":"ContainerDied","Data":"86d37e9151ad330bee33c9c89ebeeb0033c7876b4e997a9b89b3a3355eda179d"} Nov 02 18:10:27.376677 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:27.376617 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5fd3e0-77f74bdccf-xd8fb" event={"ID":"d63f9cab-764f-46ae-a522-2ef02346c21a","Type":"ContainerDied","Data":"b744e7e3952ec1ae88222ae6ab290caa3b85c4bb7cf6f9af0f4d9380a29e5168"} Nov 02 18:10:27.387967 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:27.387950 4471 scope.go:122] "RemoveContainer" containerID="6127dcc0d44863ba678229d696cb6036d9d70b5ce71298022129cca0cbdb47bc" Nov 02 18:10:27.388278 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:10:27.388247 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6127dcc0d44863ba678229d696cb6036d9d70b5ce71298022129cca0cbdb47bc\": container with ID starting with 6127dcc0d44863ba678229d696cb6036d9d70b5ce71298022129cca0cbdb47bc not found: ID does not exist" containerID="6127dcc0d44863ba678229d696cb6036d9d70b5ce71298022129cca0cbdb47bc" Nov 02 18:10:27.388352 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:27.388287 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6127dcc0d44863ba678229d696cb6036d9d70b5ce71298022129cca0cbdb47bc"} err="failed to get container status \"6127dcc0d44863ba678229d696cb6036d9d70b5ce71298022129cca0cbdb47bc\": rpc error: code = NotFound desc = could not find container \"6127dcc0d44863ba678229d696cb6036d9d70b5ce71298022129cca0cbdb47bc\": container with ID starting with 6127dcc0d44863ba678229d696cb6036d9d70b5ce71298022129cca0cbdb47bc not found: ID does not exist" Nov 02 18:10:27.388352 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:27.388331 4471 scope.go:122] "RemoveContainer" containerID="86d37e9151ad330bee33c9c89ebeeb0033c7876b4e997a9b89b3a3355eda179d" Nov 02 18:10:27.403878 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:27.403858 4471 scope.go:122] "RemoveContainer" containerID="86d37e9151ad330bee33c9c89ebeeb0033c7876b4e997a9b89b3a3355eda179d" Nov 02 18:10:27.404168 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:10:27.404148 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86d37e9151ad330bee33c9c89ebeeb0033c7876b4e997a9b89b3a3355eda179d\": container with ID starting with 86d37e9151ad330bee33c9c89ebeeb0033c7876b4e997a9b89b3a3355eda179d not found: ID does not exist" containerID="86d37e9151ad330bee33c9c89ebeeb0033c7876b4e997a9b89b3a3355eda179d" Nov 02 18:10:27.404223 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:27.404178 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86d37e9151ad330bee33c9c89ebeeb0033c7876b4e997a9b89b3a3355eda179d"} err="failed to get container status \"86d37e9151ad330bee33c9c89ebeeb0033c7876b4e997a9b89b3a3355eda179d\": rpc error: code = NotFound desc = could not find container \"86d37e9151ad330bee33c9c89ebeeb0033c7876b4e997a9b89b3a3355eda179d\": container with ID starting with 86d37e9151ad330bee33c9c89ebeeb0033c7876b4e997a9b89b3a3355eda179d not found: ID does not exist" Nov 02 18:10:27.430264 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:27.430241 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5fd3e0-77f74bdccf-xd8fb"] Nov 02 18:10:27.437004 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:27.436978 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5fd3e0-77f74bdccf-xd8fb"] Nov 02 18:10:27.469152 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:27.469126 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7pgj\" (UniqueName: \"kubernetes.io/projected/d63f9cab-764f-46ae-a522-2ef02346c21a-kube-api-access-z7pgj\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:27.473525 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:27.473500 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5fd3e0-74cfc7bbf7-cc5bk"] Nov 02 18:10:27.477774 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:27.477754 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5fd3e0-74cfc7bbf7-cc5bk"] Nov 02 18:10:28.441936 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:28.441899 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03f6f5cc-5479-478e-8a95-b86c359d2935" path="/var/lib/kubelet/pods/03f6f5cc-5479-478e-8a95-b86c359d2935/volumes" Nov 02 18:10:28.442419 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:28.442402 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c0b125a-116e-46b0-ad0b-0f7ff15fefa6" path="/var/lib/kubelet/pods/1c0b125a-116e-46b0-ad0b-0f7ff15fefa6/volumes" Nov 02 18:10:28.442989 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:28.442971 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d63f9cab-764f-46ae-a522-2ef02346c21a" path="/var/lib/kubelet/pods/d63f9cab-764f-46ae-a522-2ef02346c21a/volumes" Nov 02 18:10:29.297074 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:29.297031 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5b905-6bc9bc867-zg4xp" podUID="cbecd70b-00b8-40ac-9d1c-87043a9cc31d" containerName="payload-pre-processing" containerID="cri-o://e3ccd7b2023be19eca09a9a42f3d20b9f7dcba91913c87562613086febf52039" gracePeriod=25 Nov 02 18:10:29.590382 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:29.590357 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5b905-6bc9bc867-zg4xp" Nov 02 18:10:29.707991 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:29.707953 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbecd70b-00b8-40ac-9d1c-87043a9cc31d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbecd70b-00b8-40ac-9d1c-87043a9cc31d-plugins-config-volume\") pod \"cbecd70b-00b8-40ac-9d1c-87043a9cc31d\" (UID: \"cbecd70b-00b8-40ac-9d1c-87043a9cc31d\") " Nov 02 18:10:29.708189 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:29.708168 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbecd70b-00b8-40ac-9d1c-87043a9cc31d-kube-api-access-kmd42\" (UniqueName: \"kubernetes.io/projected/cbecd70b-00b8-40ac-9d1c-87043a9cc31d-kube-api-access-kmd42\") pod \"cbecd70b-00b8-40ac-9d1c-87043a9cc31d\" (UID: \"cbecd70b-00b8-40ac-9d1c-87043a9cc31d\") " Nov 02 18:10:29.708366 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:29.708338 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbecd70b-00b8-40ac-9d1c-87043a9cc31d-plugins-config-volume" pod "cbecd70b-00b8-40ac-9d1c-87043a9cc31d" (UID: "cbecd70b-00b8-40ac-9d1c-87043a9cc31d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:29.709018 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:29.708992 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbecd70b-00b8-40ac-9d1c-87043a9cc31d-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:29.710522 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:29.710501 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbecd70b-00b8-40ac-9d1c-87043a9cc31d-kube-api-access-kmd42" pod "cbecd70b-00b8-40ac-9d1c-87043a9cc31d" (UID: "cbecd70b-00b8-40ac-9d1c-87043a9cc31d"). InnerVolumeSpecName "kube-api-access-kmd42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:29.810533 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:29.810457 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmd42\" (UniqueName: \"kubernetes.io/projected/cbecd70b-00b8-40ac-9d1c-87043a9cc31d-kube-api-access-kmd42\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:30.406411 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:30.406372 4471 generic.go:356] "Generic (PLEG): container finished" podID="cbecd70b-00b8-40ac-9d1c-87043a9cc31d" containerID="e3ccd7b2023be19eca09a9a42f3d20b9f7dcba91913c87562613086febf52039" exitCode=0 Nov 02 18:10:30.406563 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:30.406423 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5b905-6bc9bc867-zg4xp" Nov 02 18:10:30.406563 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:30.406461 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5b905-6bc9bc867-zg4xp" event={"ID":"cbecd70b-00b8-40ac-9d1c-87043a9cc31d","Type":"ContainerDied","Data":"e3ccd7b2023be19eca09a9a42f3d20b9f7dcba91913c87562613086febf52039"} Nov 02 18:10:30.406563 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:30.406515 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d5b905-6bc9bc867-zg4xp" event={"ID":"cbecd70b-00b8-40ac-9d1c-87043a9cc31d","Type":"ContainerDied","Data":"61cb764fb4d6eb581cb748f399921f9f10d2a0595403568e9b1444a771676c99"} Nov 02 18:10:30.406563 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:30.406547 4471 scope.go:122] "RemoveContainer" containerID="e3ccd7b2023be19eca09a9a42f3d20b9f7dcba91913c87562613086febf52039" Nov 02 18:10:30.418244 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:30.418227 4471 scope.go:122] "RemoveContainer" containerID="e3ccd7b2023be19eca09a9a42f3d20b9f7dcba91913c87562613086febf52039" Nov 02 18:10:30.418539 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:10:30.418518 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3ccd7b2023be19eca09a9a42f3d20b9f7dcba91913c87562613086febf52039\": container with ID starting with e3ccd7b2023be19eca09a9a42f3d20b9f7dcba91913c87562613086febf52039 not found: ID does not exist" containerID="e3ccd7b2023be19eca09a9a42f3d20b9f7dcba91913c87562613086febf52039" Nov 02 18:10:30.418615 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:30.418550 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3ccd7b2023be19eca09a9a42f3d20b9f7dcba91913c87562613086febf52039"} err="failed to get container status \"e3ccd7b2023be19eca09a9a42f3d20b9f7dcba91913c87562613086febf52039\": rpc error: code = NotFound desc = could not find container \"e3ccd7b2023be19eca09a9a42f3d20b9f7dcba91913c87562613086febf52039\": container with ID starting with e3ccd7b2023be19eca09a9a42f3d20b9f7dcba91913c87562613086febf52039 not found: ID does not exist" Nov 02 18:10:30.476154 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:30.476116 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d5b905-6bc9bc867-zg4xp"] Nov 02 18:10:30.481525 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:30.481494 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d5b905-6bc9bc867-zg4xp"] Nov 02 18:10:31.586981 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.586939 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fbbbf8-7xz6v"] Nov 02 18:10:31.599476 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.599445 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fbbbf8-7xz6v"] Nov 02 18:10:31.808525 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.808487 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c9b1d-rl8hq"] Nov 02 18:10:31.809300 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809284 4471 cpu_manager.go:414] "containerMap: removing container" podUID="19632b35-0e41-4d90-89c3-ced789733fbc" containerName="istio-proxy" Nov 02 18:10:31.809366 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809315 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19632b35-0e41-4d90-89c3-ced789733fbc" containerName="istio-proxy" Nov 02 18:10:31.809366 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809336 4471 cpu_manager.go:414] "containerMap: removing container" podUID="19632b35-0e41-4d90-89c3-ced789733fbc" containerName="istio-proxy" Nov 02 18:10:31.809366 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809342 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19632b35-0e41-4d90-89c3-ced789733fbc" containerName="istio-proxy" Nov 02 18:10:31.809366 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809351 4471 cpu_manager.go:414] "containerMap: removing container" podUID="2bc68f8a-08a7-49b2-9564-f8041c6d96e0" containerName="payload-processing" Nov 02 18:10:31.809366 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809357 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bc68f8a-08a7-49b2-9564-f8041c6d96e0" containerName="payload-processing" Nov 02 18:10:31.809366 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809368 4471 cpu_manager.go:414] "containerMap: removing container" podUID="790f4f11-1fb0-49ae-871d-07295b20a266" containerName="istio-proxy" Nov 02 18:10:31.809543 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809375 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="790f4f11-1fb0-49ae-871d-07295b20a266" containerName="istio-proxy" Nov 02 18:10:31.809543 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809388 4471 cpu_manager.go:414] "containerMap: removing container" podUID="2a58f095-38b2-45e4-8cd0-1a2136540bbd" containerName="revoke-keys" Nov 02 18:10:31.809543 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809393 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a58f095-38b2-45e4-8cd0-1a2136540bbd" containerName="revoke-keys" Nov 02 18:10:31.809543 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809401 4471 cpu_manager.go:414] "containerMap: removing container" podUID="1c0b125a-116e-46b0-ad0b-0f7ff15fefa6" containerName="payload-processing" Nov 02 18:10:31.809543 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809406 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c0b125a-116e-46b0-ad0b-0f7ff15fefa6" containerName="payload-processing" Nov 02 18:10:31.809543 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809416 4471 cpu_manager.go:414] "containerMap: removing container" podUID="aaca053e-d726-46af-a659-2a15071343c7" containerName="istio-proxy" Nov 02 18:10:31.809543 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809422 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaca053e-d726-46af-a659-2a15071343c7" containerName="istio-proxy" Nov 02 18:10:31.809543 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809435 4471 cpu_manager.go:414] "containerMap: removing container" podUID="03f6f5cc-5479-478e-8a95-b86c359d2935" containerName="payload-pre-processing" Nov 02 18:10:31.809543 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809440 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03f6f5cc-5479-478e-8a95-b86c359d2935" containerName="payload-pre-processing" Nov 02 18:10:31.809543 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809453 4471 cpu_manager.go:414] "containerMap: removing container" podUID="0b34a701-da68-40d6-9bd1-bcdd3a3e2866" containerName="revoke-keys" Nov 02 18:10:31.809543 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809457 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b34a701-da68-40d6-9bd1-bcdd3a3e2866" containerName="revoke-keys" Nov 02 18:10:31.809543 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809468 4471 cpu_manager.go:414] "containerMap: removing container" podUID="cbecd70b-00b8-40ac-9d1c-87043a9cc31d" containerName="payload-pre-processing" Nov 02 18:10:31.809543 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809473 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbecd70b-00b8-40ac-9d1c-87043a9cc31d" containerName="payload-pre-processing" Nov 02 18:10:31.809543 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809493 4471 cpu_manager.go:414] "containerMap: removing container" podUID="6183d0fa-f089-47f0-97a0-163160fc9f25" containerName="istio-proxy" Nov 02 18:10:31.809543 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809498 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6183d0fa-f089-47f0-97a0-163160fc9f25" containerName="istio-proxy" Nov 02 18:10:31.809543 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809509 4471 cpu_manager.go:414] "containerMap: removing container" podUID="d63f9cab-764f-46ae-a522-2ef02346c21a" containerName="payload-pre-processing" Nov 02 18:10:31.809543 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809514 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d63f9cab-764f-46ae-a522-2ef02346c21a" containerName="payload-pre-processing" Nov 02 18:10:31.809543 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809523 4471 cpu_manager.go:414] "containerMap: removing container" podUID="4c0f219b-fdac-4b79-92fc-80ae017a205a" containerName="revoke-keys" Nov 02 18:10:31.809543 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809528 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c0f219b-fdac-4b79-92fc-80ae017a205a" containerName="revoke-keys" Nov 02 18:10:31.809543 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809536 4471 cpu_manager.go:414] "containerMap: removing container" podUID="f6a3d6e8-b54f-48ee-9381-4ee92f76e158" containerName="revoke-keys" Nov 02 18:10:31.809543 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809541 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6a3d6e8-b54f-48ee-9381-4ee92f76e158" containerName="revoke-keys" Nov 02 18:10:31.809543 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809550 4471 cpu_manager.go:414] "containerMap: removing container" podUID="c1e9cee8-e915-4a1f-819d-226dcda7485a" containerName="istio-proxy" Nov 02 18:10:31.810173 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809555 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1e9cee8-e915-4a1f-819d-226dcda7485a" containerName="istio-proxy" Nov 02 18:10:31.810173 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809565 4471 cpu_manager.go:414] "containerMap: removing container" podUID="e304caf1-38c3-4cbb-9c2f-db257e76ada7" containerName="istio-proxy" Nov 02 18:10:31.810173 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809570 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e304caf1-38c3-4cbb-9c2f-db257e76ada7" containerName="istio-proxy" Nov 02 18:10:31.810173 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809685 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="19632b35-0e41-4d90-89c3-ced789733fbc" containerName="istio-proxy" Nov 02 18:10:31.810173 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809691 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bc68f8a-08a7-49b2-9564-f8041c6d96e0" containerName="payload-processing" Nov 02 18:10:31.810173 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809701 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="790f4f11-1fb0-49ae-871d-07295b20a266" containerName="istio-proxy" Nov 02 18:10:31.810173 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809710 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a58f095-38b2-45e4-8cd0-1a2136540bbd" containerName="revoke-keys" Nov 02 18:10:31.810173 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809717 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c0b125a-116e-46b0-ad0b-0f7ff15fefa6" containerName="payload-processing" Nov 02 18:10:31.810173 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809725 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaca053e-d726-46af-a659-2a15071343c7" containerName="istio-proxy" Nov 02 18:10:31.810173 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809734 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="03f6f5cc-5479-478e-8a95-b86c359d2935" containerName="payload-pre-processing" Nov 02 18:10:31.810173 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809741 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b34a701-da68-40d6-9bd1-bcdd3a3e2866" containerName="revoke-keys" Nov 02 18:10:31.810173 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809749 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbecd70b-00b8-40ac-9d1c-87043a9cc31d" containerName="payload-pre-processing" Nov 02 18:10:31.810173 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809757 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="6183d0fa-f089-47f0-97a0-163160fc9f25" containerName="istio-proxy" Nov 02 18:10:31.810173 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809764 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="d63f9cab-764f-46ae-a522-2ef02346c21a" containerName="payload-pre-processing" Nov 02 18:10:31.810173 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809774 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c0f219b-fdac-4b79-92fc-80ae017a205a" containerName="revoke-keys" Nov 02 18:10:31.810173 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809781 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6a3d6e8-b54f-48ee-9381-4ee92f76e158" containerName="revoke-keys" Nov 02 18:10:31.810173 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809788 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1e9cee8-e915-4a1f-819d-226dcda7485a" containerName="istio-proxy" Nov 02 18:10:31.810173 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.809794 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="e304caf1-38c3-4cbb-9c2f-db257e76ada7" containerName="istio-proxy" Nov 02 18:10:31.813384 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.813365 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c9b1d-rl8hq" Nov 02 18:10:31.829031 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.829010 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:10:31.829137 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.829119 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 18:10:31.841035 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.840976 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c9b1d-rl8hq"] Nov 02 18:10:31.870644 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.870609 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqrq8\" (UniqueName: \"kubernetes.io/projected/5a3f2721-ff4f-4b04-a041-76b3096534e9-kube-api-access-wqrq8\") pod \"maas-api-revoke-keys-e2e-inf-b-2c9b1d-rl8hq\" (UID: \"5a3f2721-ff4f-4b04-a041-76b3096534e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c9b1d-rl8hq" Nov 02 18:10:31.870877 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.870851 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a3f2721-ff4f-4b04-a041-76b3096534e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2c9b1d-rl8hq\" (UID: \"5a3f2721-ff4f-4b04-a041-76b3096534e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c9b1d-rl8hq" Nov 02 18:10:31.870982 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.870952 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a3f2721-ff4f-4b04-a041-76b3096534e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2c9b1d-rl8hq\" (UID: \"5a3f2721-ff4f-4b04-a041-76b3096534e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c9b1d-rl8hq" Nov 02 18:10:31.972219 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.972182 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a3f2721-ff4f-4b04-a041-76b3096534e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2c9b1d-rl8hq\" (UID: \"5a3f2721-ff4f-4b04-a041-76b3096534e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c9b1d-rl8hq" Nov 02 18:10:31.972435 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.972414 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqrq8\" (UniqueName: \"kubernetes.io/projected/5a3f2721-ff4f-4b04-a041-76b3096534e9-kube-api-access-wqrq8\") pod \"maas-api-revoke-keys-e2e-inf-b-2c9b1d-rl8hq\" (UID: \"5a3f2721-ff4f-4b04-a041-76b3096534e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c9b1d-rl8hq" Nov 02 18:10:31.972637 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.972618 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a3f2721-ff4f-4b04-a041-76b3096534e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2c9b1d-rl8hq\" (UID: \"5a3f2721-ff4f-4b04-a041-76b3096534e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c9b1d-rl8hq" Nov 02 18:10:31.973231 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.973205 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a3f2721-ff4f-4b04-a041-76b3096534e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2c9b1d-rl8hq\" (UID: \"5a3f2721-ff4f-4b04-a041-76b3096534e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c9b1d-rl8hq" Nov 02 18:10:31.974647 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.974631 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a3f2721-ff4f-4b04-a041-76b3096534e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2c9b1d-rl8hq\" (UID: \"5a3f2721-ff4f-4b04-a041-76b3096534e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c9b1d-rl8hq" Nov 02 18:10:31.996138 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:31.996114 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqrq8\" (UniqueName: \"kubernetes.io/projected/5a3f2721-ff4f-4b04-a041-76b3096534e9-kube-api-access-wqrq8\") pod \"maas-api-revoke-keys-e2e-inf-b-2c9b1d-rl8hq\" (UID: \"5a3f2721-ff4f-4b04-a041-76b3096534e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c9b1d-rl8hq" Nov 02 18:10:32.127399 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:32.127316 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c9b1d-rl8hq" Nov 02 18:10:32.431458 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:10:32.431429 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a3f2721_ff4f_4b04_a041_76b3096534e9.slice/crio-8479bb7d767fa17d94bfde3b4183690e67b3f64afffd5a86f4a25e3ef730f66a WatchSource:0}: Error finding container 8479bb7d767fa17d94bfde3b4183690e67b3f64afffd5a86f4a25e3ef730f66a: Status 404 returned error can't find the container with id 8479bb7d767fa17d94bfde3b4183690e67b3f64afffd5a86f4a25e3ef730f66a Nov 02 18:10:32.444161 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:32.444133 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a58f095-38b2-45e4-8cd0-1a2136540bbd" path="/var/lib/kubelet/pods/2a58f095-38b2-45e4-8cd0-1a2136540bbd/volumes" Nov 02 18:10:32.444621 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:32.444604 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbecd70b-00b8-40ac-9d1c-87043a9cc31d" path="/var/lib/kubelet/pods/cbecd70b-00b8-40ac-9d1c-87043a9cc31d/volumes" Nov 02 18:10:32.444954 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:32.444937 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c9b1d-rl8hq"] Nov 02 18:10:33.433221 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:33.433176 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c9b1d-rl8hq" event={"ID":"5a3f2721-ff4f-4b04-a041-76b3096534e9","Type":"ContainerStarted","Data":"ceafb8872240698414d76dea62da1717f0ae739b460a91d9eba0e22f710479e5"} Nov 02 18:10:33.433723 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:33.433233 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c9b1d-rl8hq" event={"ID":"5a3f2721-ff4f-4b04-a041-76b3096534e9","Type":"ContainerStarted","Data":"8479bb7d767fa17d94bfde3b4183690e67b3f64afffd5a86f4a25e3ef730f66a"} Nov 02 18:10:34.372575 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:34.372518 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c9b1d-rl8hq" podStartSLOduration=3.372498691 podStartE2EDuration="3.372498691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:10:33.485197605 +0000 UTC m=+2451.180553080" watchObservedRunningTime="2026-11-02 18:10:34.372498691 +0000 UTC m=+2452.067854164" Nov 02 18:10:34.374450 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:34.374420 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2fc9f-ddbq6"] Nov 02 18:10:34.386419 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:34.386396 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2fc9f-ddbq6"] Nov 02 18:10:34.442966 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:34.442931 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c0f219b-fdac-4b79-92fc-80ae017a205a" path="/var/lib/kubelet/pods/4c0f219b-fdac-4b79-92fc-80ae017a205a/volumes" Nov 02 18:10:34.443442 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:34.443246 4471 generic.go:356] "Generic (PLEG): container finished" podID="5a3f2721-ff4f-4b04-a041-76b3096534e9" containerID="ceafb8872240698414d76dea62da1717f0ae739b460a91d9eba0e22f710479e5" exitCode=0 Nov 02 18:10:34.443442 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:34.443318 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c9b1d-rl8hq" event={"ID":"5a3f2721-ff4f-4b04-a041-76b3096534e9","Type":"ContainerDied","Data":"ceafb8872240698414d76dea62da1717f0ae739b460a91d9eba0e22f710479e5"} Nov 02 18:10:35.616691 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:35.616666 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c9b1d-rl8hq" Nov 02 18:10:35.681777 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:35.681741 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a3f2721-ff4f-4b04-a041-76b3096534e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a3f2721-ff4f-4b04-a041-76b3096534e9-maas-api-xks-ca\") pod \"5a3f2721-ff4f-4b04-a041-76b3096534e9\" (UID: \"5a3f2721-ff4f-4b04-a041-76b3096534e9\") " Nov 02 18:10:35.682075 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:35.682049 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a3f2721-ff4f-4b04-a041-76b3096534e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a3f2721-ff4f-4b04-a041-76b3096534e9-maas-api-openshift-ca\") pod \"5a3f2721-ff4f-4b04-a041-76b3096534e9\" (UID: \"5a3f2721-ff4f-4b04-a041-76b3096534e9\") " Nov 02 18:10:35.682214 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:35.682196 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a3f2721-ff4f-4b04-a041-76b3096534e9-kube-api-access-wqrq8\" (UniqueName: \"kubernetes.io/projected/5a3f2721-ff4f-4b04-a041-76b3096534e9-kube-api-access-wqrq8\") pod \"5a3f2721-ff4f-4b04-a041-76b3096534e9\" (UID: \"5a3f2721-ff4f-4b04-a041-76b3096534e9\") " Nov 02 18:10:35.682489 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:35.682460 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a3f2721-ff4f-4b04-a041-76b3096534e9-maas-api-openshift-ca" pod "5a3f2721-ff4f-4b04-a041-76b3096534e9" (UID: "5a3f2721-ff4f-4b04-a041-76b3096534e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:35.683106 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:35.683080 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a3f2721-ff4f-4b04-a041-76b3096534e9-maas-api-openshift-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:35.684500 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:35.684474 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a3f2721-ff4f-4b04-a041-76b3096534e9-maas-api-xks-ca" pod "5a3f2721-ff4f-4b04-a041-76b3096534e9" (UID: "5a3f2721-ff4f-4b04-a041-76b3096534e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:10:35.685180 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:35.685155 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a3f2721-ff4f-4b04-a041-76b3096534e9-kube-api-access-wqrq8" pod "5a3f2721-ff4f-4b04-a041-76b3096534e9" (UID: "5a3f2721-ff4f-4b04-a041-76b3096534e9"). InnerVolumeSpecName "kube-api-access-wqrq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:35.784544 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:35.784502 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqrq8\" (UniqueName: \"kubernetes.io/projected/5a3f2721-ff4f-4b04-a041-76b3096534e9-kube-api-access-wqrq8\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:35.784544 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:35.784534 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a3f2721-ff4f-4b04-a041-76b3096534e9-maas-api-xks-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:10:36.459694 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:36.459671 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c9b1d-rl8hq" Nov 02 18:10:36.459855 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:36.459720 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c9b1d-rl8hq" event={"ID":"5a3f2721-ff4f-4b04-a041-76b3096534e9","Type":"ContainerDied","Data":"8479bb7d767fa17d94bfde3b4183690e67b3f64afffd5a86f4a25e3ef730f66a"} Nov 02 18:10:36.459855 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:10:36.459773 4471 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8479bb7d767fa17d94bfde3b4183690e67b3f64afffd5a86f4a25e3ef730f66a" Nov 02 18:10:39.185702 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:10:39.185658 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:11:00.239558 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:00.239481 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5fd3e0-hkdvc"] Nov 02 18:11:00.278940 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:00.278901 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5fd3e0-hkdvc"] Nov 02 18:11:00.443771 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:00.443737 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6a3d6e8-b54f-48ee-9381-4ee92f76e158" path="/var/lib/kubelet/pods/f6a3d6e8-b54f-48ee-9381-4ee92f76e158/volumes" Nov 02 18:11:03.027326 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:03.014843 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5b905-fvz8z"] Nov 02 18:11:03.041414 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:03.038144 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5b905-fvz8z"] Nov 02 18:11:04.443004 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:04.442969 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b34a701-da68-40d6-9bd1-bcdd3a3e2866" path="/var/lib/kubelet/pods/0b34a701-da68-40d6-9bd1-bcdd3a3e2866/volumes" Nov 02 18:11:04.616533 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:04.616496 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj"] Nov 02 18:11:04.616812 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:04.616766 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj" podUID="a186e1c4-6552-487a-bb08-3bd479c1b1bf" containerName="maas-api" containerID="cri-o://6acb8d9f617be5f7080e9de7d6e7dde2ec18aeb1ce196561aec72d6e4f3e3502" gracePeriod=30 Nov 02 18:11:04.771404 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:04.771365 4471 generic.go:356] "Generic (PLEG): container finished" podID="a186e1c4-6552-487a-bb08-3bd479c1b1bf" containerID="6acb8d9f617be5f7080e9de7d6e7dde2ec18aeb1ce196561aec72d6e4f3e3502" exitCode=0 Nov 02 18:11:04.771561 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:04.771430 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj" event={"ID":"a186e1c4-6552-487a-bb08-3bd479c1b1bf","Type":"ContainerDied","Data":"6acb8d9f617be5f7080e9de7d6e7dde2ec18aeb1ce196561aec72d6e4f3e3502"} Nov 02 18:11:05.346008 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:05.345975 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj" Nov 02 18:11:05.474369 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:05.474252 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a186e1c4-6552-487a-bb08-3bd479c1b1bf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a186e1c4-6552-487a-bb08-3bd479c1b1bf-openshift-service-ca\") pod \"a186e1c4-6552-487a-bb08-3bd479c1b1bf\" (UID: \"a186e1c4-6552-487a-bb08-3bd479c1b1bf\") " Nov 02 18:11:05.474837 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:05.474432 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a186e1c4-6552-487a-bb08-3bd479c1b1bf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a186e1c4-6552-487a-bb08-3bd479c1b1bf-metrics-certs\") pod \"a186e1c4-6552-487a-bb08-3bd479c1b1bf\" (UID: \"a186e1c4-6552-487a-bb08-3bd479c1b1bf\") " Nov 02 18:11:05.474837 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:05.474618 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a186e1c4-6552-487a-bb08-3bd479c1b1bf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a186e1c4-6552-487a-bb08-3bd479c1b1bf-maas-api-tls\") pod \"a186e1c4-6552-487a-bb08-3bd479c1b1bf\" (UID: \"a186e1c4-6552-487a-bb08-3bd479c1b1bf\") " Nov 02 18:11:05.474837 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:05.474646 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a186e1c4-6552-487a-bb08-3bd479c1b1bf-kube-api-access-r5ggp\" (UniqueName: \"kubernetes.io/projected/a186e1c4-6552-487a-bb08-3bd479c1b1bf-kube-api-access-r5ggp\") pod \"a186e1c4-6552-487a-bb08-3bd479c1b1bf\" (UID: \"a186e1c4-6552-487a-bb08-3bd479c1b1bf\") " Nov 02 18:11:05.474837 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:05.474691 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a186e1c4-6552-487a-bb08-3bd479c1b1bf-openshift-service-ca" pod "a186e1c4-6552-487a-bb08-3bd479c1b1bf" (UID: "a186e1c4-6552-487a-bb08-3bd479c1b1bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:05.475248 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:05.475228 4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a186e1c4-6552-487a-bb08-3bd479c1b1bf-openshift-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:05.476656 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:05.476627 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a186e1c4-6552-487a-bb08-3bd479c1b1bf-kube-api-access-r5ggp" pod "a186e1c4-6552-487a-bb08-3bd479c1b1bf" (UID: "a186e1c4-6552-487a-bb08-3bd479c1b1bf"). InnerVolumeSpecName "kube-api-access-r5ggp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:05.476765 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:05.476628 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a186e1c4-6552-487a-bb08-3bd479c1b1bf-metrics-certs" pod "a186e1c4-6552-487a-bb08-3bd479c1b1bf" (UID: "a186e1c4-6552-487a-bb08-3bd479c1b1bf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:11:05.476765 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:05.476669 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a186e1c4-6552-487a-bb08-3bd479c1b1bf-maas-api-tls" pod "a186e1c4-6552-487a-bb08-3bd479c1b1bf" (UID: "a186e1c4-6552-487a-bb08-3bd479c1b1bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:11:05.577365 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:05.577331 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a186e1c4-6552-487a-bb08-3bd479c1b1bf-maas-api-tls\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:05.577365 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:05.577367 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5ggp\" (UniqueName: \"kubernetes.io/projected/a186e1c4-6552-487a-bb08-3bd479c1b1bf-kube-api-access-r5ggp\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:05.577572 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:05.577378 4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a186e1c4-6552-487a-bb08-3bd479c1b1bf-metrics-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:05.676388 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:05.676350 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c9b1d-rl8hq"] Nov 02 18:11:05.684752 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:05.684718 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c9b1d-rl8hq"] Nov 02 18:11:05.820327 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:05.817632 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj" event={"ID":"a186e1c4-6552-487a-bb08-3bd479c1b1bf","Type":"ContainerDied","Data":"d3133b1de006acac30651ec49b1a0e910b7cb13361db8dbf443c65dc310da001"} Nov 02 18:11:05.820327 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:05.817679 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj" Nov 02 18:11:05.820327 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:05.817745 4471 scope.go:122] "RemoveContainer" containerID="6acb8d9f617be5f7080e9de7d6e7dde2ec18aeb1ce196561aec72d6e4f3e3502" Nov 02 18:11:05.876456 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:05.876427 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj"] Nov 02 18:11:05.883424 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:05.883395 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2c9b1d-5d69ff4958-57ppj"] Nov 02 18:11:06.442771 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:06.442724 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a3f2721-ff4f-4b04-a041-76b3096534e9" path="/var/lib/kubelet/pods/5a3f2721-ff4f-4b04-a041-76b3096534e9/volumes" Nov 02 18:11:06.443592 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:06.443561 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a186e1c4-6552-487a-bb08-3bd479c1b1bf" path="/var/lib/kubelet/pods/a186e1c4-6552-487a-bb08-3bd479c1b1bf/volumes" Nov 02 18:11:09.185135 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:11:09.185099 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:11:16.005871 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.005816 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-56665b75dc-whm5m"] Nov 02 18:11:16.007166 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.007140 4471 cpu_manager.go:414] "containerMap: removing container" podUID="5a3f2721-ff4f-4b04-a041-76b3096534e9" containerName="revoke-keys" Nov 02 18:11:16.007292 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.007169 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a3f2721-ff4f-4b04-a041-76b3096534e9" containerName="revoke-keys" Nov 02 18:11:16.007292 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.007216 4471 cpu_manager.go:414] "containerMap: removing container" podUID="a186e1c4-6552-487a-bb08-3bd479c1b1bf" containerName="maas-api" Nov 02 18:11:16.007292 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.007225 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a186e1c4-6552-487a-bb08-3bd479c1b1bf" containerName="maas-api" Nov 02 18:11:16.007493 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.007419 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="19632b35-0e41-4d90-89c3-ced789733fbc" containerName="istio-proxy" Nov 02 18:11:16.007493 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.007437 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a3f2721-ff4f-4b04-a041-76b3096534e9" containerName="revoke-keys" Nov 02 18:11:16.007493 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.007453 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="a186e1c4-6552-487a-bb08-3bd479c1b1bf" containerName="maas-api" Nov 02 18:11:16.012142 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.011434 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56665b75dc-whm5m" Nov 02 18:11:16.047050 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.047020 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56665b75dc-whm5m"] Nov 02 18:11:16.100371 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.092346 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cde2c4ed-f0d0-4c35-8b75-8a8117bfb819-console-config\") pod \"console-56665b75dc-whm5m\" (UID: \"cde2c4ed-f0d0-4c35-8b75-8a8117bfb819\") " pod="openshift-console/console-56665b75dc-whm5m" Nov 02 18:11:16.100371 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.092413 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cde2c4ed-f0d0-4c35-8b75-8a8117bfb819-oauth-serving-cert\") pod \"console-56665b75dc-whm5m\" (UID: \"cde2c4ed-f0d0-4c35-8b75-8a8117bfb819\") " pod="openshift-console/console-56665b75dc-whm5m" Nov 02 18:11:16.100371 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.092447 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvmfj\" (UniqueName: \"kubernetes.io/projected/cde2c4ed-f0d0-4c35-8b75-8a8117bfb819-kube-api-access-rvmfj\") pod \"console-56665b75dc-whm5m\" (UID: \"cde2c4ed-f0d0-4c35-8b75-8a8117bfb819\") " pod="openshift-console/console-56665b75dc-whm5m" Nov 02 18:11:16.100371 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.092557 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cde2c4ed-f0d0-4c35-8b75-8a8117bfb819-console-serving-cert\") pod \"console-56665b75dc-whm5m\" (UID: \"cde2c4ed-f0d0-4c35-8b75-8a8117bfb819\") " pod="openshift-console/console-56665b75dc-whm5m" Nov 02 18:11:16.100371 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.093061 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cde2c4ed-f0d0-4c35-8b75-8a8117bfb819-console-oauth-config\") pod \"console-56665b75dc-whm5m\" (UID: \"cde2c4ed-f0d0-4c35-8b75-8a8117bfb819\") " pod="openshift-console/console-56665b75dc-whm5m" Nov 02 18:11:16.100371 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.093160 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cde2c4ed-f0d0-4c35-8b75-8a8117bfb819-tmp\") pod \"console-56665b75dc-whm5m\" (UID: \"cde2c4ed-f0d0-4c35-8b75-8a8117bfb819\") " pod="openshift-console/console-56665b75dc-whm5m" Nov 02 18:11:16.100371 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.093288 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cde2c4ed-f0d0-4c35-8b75-8a8117bfb819-service-ca\") pod \"console-56665b75dc-whm5m\" (UID: \"cde2c4ed-f0d0-4c35-8b75-8a8117bfb819\") " pod="openshift-console/console-56665b75dc-whm5m" Nov 02 18:11:16.100371 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.093363 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cde2c4ed-f0d0-4c35-8b75-8a8117bfb819-trusted-ca-bundle\") pod \"console-56665b75dc-whm5m\" (UID: \"cde2c4ed-f0d0-4c35-8b75-8a8117bfb819\") " pod="openshift-console/console-56665b75dc-whm5m" Nov 02 18:11:16.194846 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.194799 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cde2c4ed-f0d0-4c35-8b75-8a8117bfb819-console-serving-cert\") pod \"console-56665b75dc-whm5m\" (UID: \"cde2c4ed-f0d0-4c35-8b75-8a8117bfb819\") " pod="openshift-console/console-56665b75dc-whm5m" Nov 02 18:11:16.195064 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.194955 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cde2c4ed-f0d0-4c35-8b75-8a8117bfb819-console-oauth-config\") pod \"console-56665b75dc-whm5m\" (UID: \"cde2c4ed-f0d0-4c35-8b75-8a8117bfb819\") " pod="openshift-console/console-56665b75dc-whm5m" Nov 02 18:11:16.195064 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.195042 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cde2c4ed-f0d0-4c35-8b75-8a8117bfb819-tmp\") pod \"console-56665b75dc-whm5m\" (UID: \"cde2c4ed-f0d0-4c35-8b75-8a8117bfb819\") " pod="openshift-console/console-56665b75dc-whm5m" Nov 02 18:11:16.195207 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.195091 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cde2c4ed-f0d0-4c35-8b75-8a8117bfb819-service-ca\") pod \"console-56665b75dc-whm5m\" (UID: \"cde2c4ed-f0d0-4c35-8b75-8a8117bfb819\") " pod="openshift-console/console-56665b75dc-whm5m" Nov 02 18:11:16.195207 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.195146 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cde2c4ed-f0d0-4c35-8b75-8a8117bfb819-trusted-ca-bundle\") pod \"console-56665b75dc-whm5m\" (UID: \"cde2c4ed-f0d0-4c35-8b75-8a8117bfb819\") " pod="openshift-console/console-56665b75dc-whm5m" Nov 02 18:11:16.195360 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.195283 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cde2c4ed-f0d0-4c35-8b75-8a8117bfb819-console-config\") pod \"console-56665b75dc-whm5m\" (UID: \"cde2c4ed-f0d0-4c35-8b75-8a8117bfb819\") " pod="openshift-console/console-56665b75dc-whm5m" Nov 02 18:11:16.195360 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.195322 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cde2c4ed-f0d0-4c35-8b75-8a8117bfb819-oauth-serving-cert\") pod \"console-56665b75dc-whm5m\" (UID: \"cde2c4ed-f0d0-4c35-8b75-8a8117bfb819\") " pod="openshift-console/console-56665b75dc-whm5m" Nov 02 18:11:16.195360 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.195348 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvmfj\" (UniqueName: \"kubernetes.io/projected/cde2c4ed-f0d0-4c35-8b75-8a8117bfb819-kube-api-access-rvmfj\") pod \"console-56665b75dc-whm5m\" (UID: \"cde2c4ed-f0d0-4c35-8b75-8a8117bfb819\") " pod="openshift-console/console-56665b75dc-whm5m" Nov 02 18:11:16.195528 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.195507 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cde2c4ed-f0d0-4c35-8b75-8a8117bfb819-tmp\") pod \"console-56665b75dc-whm5m\" (UID: \"cde2c4ed-f0d0-4c35-8b75-8a8117bfb819\") " pod="openshift-console/console-56665b75dc-whm5m" Nov 02 18:11:16.196131 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.196102 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cde2c4ed-f0d0-4c35-8b75-8a8117bfb819-service-ca\") pod \"console-56665b75dc-whm5m\" (UID: \"cde2c4ed-f0d0-4c35-8b75-8a8117bfb819\") " pod="openshift-console/console-56665b75dc-whm5m" Nov 02 18:11:16.196255 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.196156 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cde2c4ed-f0d0-4c35-8b75-8a8117bfb819-console-config\") pod \"console-56665b75dc-whm5m\" (UID: \"cde2c4ed-f0d0-4c35-8b75-8a8117bfb819\") " pod="openshift-console/console-56665b75dc-whm5m" Nov 02 18:11:16.196393 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.196364 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cde2c4ed-f0d0-4c35-8b75-8a8117bfb819-oauth-serving-cert\") pod \"console-56665b75dc-whm5m\" (UID: \"cde2c4ed-f0d0-4c35-8b75-8a8117bfb819\") " pod="openshift-console/console-56665b75dc-whm5m" Nov 02 18:11:16.196393 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.196379 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cde2c4ed-f0d0-4c35-8b75-8a8117bfb819-trusted-ca-bundle\") pod \"console-56665b75dc-whm5m\" (UID: \"cde2c4ed-f0d0-4c35-8b75-8a8117bfb819\") " pod="openshift-console/console-56665b75dc-whm5m" Nov 02 18:11:16.197753 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.197725 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cde2c4ed-f0d0-4c35-8b75-8a8117bfb819-console-oauth-config\") pod \"console-56665b75dc-whm5m\" (UID: \"cde2c4ed-f0d0-4c35-8b75-8a8117bfb819\") " pod="openshift-console/console-56665b75dc-whm5m" Nov 02 18:11:16.197941 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.197926 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cde2c4ed-f0d0-4c35-8b75-8a8117bfb819-console-serving-cert\") pod \"console-56665b75dc-whm5m\" (UID: \"cde2c4ed-f0d0-4c35-8b75-8a8117bfb819\") " pod="openshift-console/console-56665b75dc-whm5m" Nov 02 18:11:16.226873 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.226840 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvmfj\" (UniqueName: \"kubernetes.io/projected/cde2c4ed-f0d0-4c35-8b75-8a8117bfb819-kube-api-access-rvmfj\") pod \"console-56665b75dc-whm5m\" (UID: \"cde2c4ed-f0d0-4c35-8b75-8a8117bfb819\") " pod="openshift-console/console-56665b75dc-whm5m" Nov 02 18:11:16.336427 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.336389 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56665b75dc-whm5m" Nov 02 18:11:16.652361 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:11:16.652330 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde2c4ed_f0d0_4c35_8b75_8a8117bfb819.slice/crio-c915ea7c91cf384b945475b169dd3273d0aaae8ec94238f4f0abab9eb622686b WatchSource:0}: Error finding container c915ea7c91cf384b945475b169dd3273d0aaae8ec94238f4f0abab9eb622686b: Status 404 returned error can't find the container with id c915ea7c91cf384b945475b169dd3273d0aaae8ec94238f4f0abab9eb622686b Nov 02 18:11:16.653501 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.653469 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56665b75dc-whm5m"] Nov 02 18:11:16.924407 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.924279 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56665b75dc-whm5m" event={"ID":"cde2c4ed-f0d0-4c35-8b75-8a8117bfb819","Type":"ContainerStarted","Data":"2bcf8fb9a6199650e0f9a11fb818f46e5ccf9f83e3c710e7be5b41f2d2d30c51"} Nov 02 18:11:16.924407 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.924356 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56665b75dc-whm5m" event={"ID":"cde2c4ed-f0d0-4c35-8b75-8a8117bfb819","Type":"ContainerStarted","Data":"c915ea7c91cf384b945475b169dd3273d0aaae8ec94238f4f0abab9eb622686b"} Nov 02 18:11:16.976139 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:16.976082 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-56665b75dc-whm5m" podStartSLOduration=1.976066187 podStartE2EDuration="1.976066187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:11:16.974652708 +0000 UTC m=+2494.670008181" watchObservedRunningTime="2026-11-02 18:11:16.976066187 +0000 UTC m=+2494.671421662" Nov 02 18:11:26.337375 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:26.337335 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56665b75dc-whm5m" Nov 02 18:11:26.337782 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:26.337415 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56665b75dc-whm5m" Nov 02 18:11:26.343170 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:26.343144 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56665b75dc-whm5m" Nov 02 18:11:27.013213 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:27.013179 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56665b75dc-whm5m" Nov 02 18:11:27.123539 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:27.123504 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9b8d5bbf-xnjnp"] Nov 02 18:11:33.689011 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:33.688969 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b"] Nov 02 18:11:33.689515 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:33.689225 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" podUID="341883ea-c8c4-49dd-ba87-ef0a3a539065" containerName="istio-proxy" containerID="cri-o://223b0b7f02faec9930397bf9dd171da4d453cca519329ca75cff4e1ce02d36de" gracePeriod=30 Nov 02 18:11:38.452093 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:38.452054 4471 patch_prober.go:29] interesting pod/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:11:38.452661 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:38.452115 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" podUID="341883ea-c8c4-49dd-ba87-ef0a3a539065" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:11:39.005998 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.005969 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 18:11:39.089526 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.089490 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9"] Nov 02 18:11:39.089753 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.089727 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" podUID="38fea887-fcc0-42fa-a1c9-320e44f70562" containerName="istio-proxy" containerID="cri-o://2f33607f9e421093ec91c99de5a98130292399afa5ee9d6fa12df8e0e7e23650" gracePeriod=30 Nov 02 18:11:39.094912 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.094877 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-istio-envoy\") pod \"341883ea-c8c4-49dd-ba87-ef0a3a539065\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " Nov 02 18:11:39.095076 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.094920 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/341883ea-c8c4-49dd-ba87-ef0a3a539065-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/341883ea-c8c4-49dd-ba87-ef0a3a539065-istiod-ca-cert\") pod \"341883ea-c8c4-49dd-ba87-ef0a3a539065\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " Nov 02 18:11:39.095076 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.094967 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-istio-data\") pod \"341883ea-c8c4-49dd-ba87-ef0a3a539065\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " Nov 02 18:11:39.095076 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.095003 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-workload-certs\") pod \"341883ea-c8c4-49dd-ba87-ef0a3a539065\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " Nov 02 18:11:39.095293 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.095086 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/341883ea-c8c4-49dd-ba87-ef0a3a539065-istio-token\" (UniqueName: \"kubernetes.io/projected/341883ea-c8c4-49dd-ba87-ef0a3a539065-istio-token\") pod \"341883ea-c8c4-49dd-ba87-ef0a3a539065\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " Nov 02 18:11:39.095293 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.095142 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-workload-socket\") pod \"341883ea-c8c4-49dd-ba87-ef0a3a539065\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " Nov 02 18:11:39.095430 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.095413 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/341883ea-c8c4-49dd-ba87-ef0a3a539065-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/341883ea-c8c4-49dd-ba87-ef0a3a539065-istio-podinfo\") pod \"341883ea-c8c4-49dd-ba87-ef0a3a539065\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " Nov 02 18:11:39.095495 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.095481 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/341883ea-c8c4-49dd-ba87-ef0a3a539065-kube-api-access-229bb\" (UniqueName: \"kubernetes.io/projected/341883ea-c8c4-49dd-ba87-ef0a3a539065-kube-api-access-229bb\") pod \"341883ea-c8c4-49dd-ba87-ef0a3a539065\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " Nov 02 18:11:39.095567 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.095545 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-credential-socket\") pod \"341883ea-c8c4-49dd-ba87-ef0a3a539065\" (UID: \"341883ea-c8c4-49dd-ba87-ef0a3a539065\") " Nov 02 18:11:39.096195 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.096146 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/341883ea-c8c4-49dd-ba87-ef0a3a539065-istiod-ca-cert" pod "341883ea-c8c4-49dd-ba87-ef0a3a539065" (UID: "341883ea-c8c4-49dd-ba87-ef0a3a539065"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:39.096510 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.096482 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-credential-socket" pod "341883ea-c8c4-49dd-ba87-ef0a3a539065" (UID: "341883ea-c8c4-49dd-ba87-ef0a3a539065"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:39.096678 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.096661 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:39.096756 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.096678 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/341883ea-c8c4-49dd-ba87-ef0a3a539065-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:39.096970 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.096932 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-istio-data" pod "341883ea-c8c4-49dd-ba87-ef0a3a539065" (UID: "341883ea-c8c4-49dd-ba87-ef0a3a539065"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:39.097147 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.097122 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-workload-socket" pod "341883ea-c8c4-49dd-ba87-ef0a3a539065" (UID: "341883ea-c8c4-49dd-ba87-ef0a3a539065"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:39.097798 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.097735 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-workload-certs" pod "341883ea-c8c4-49dd-ba87-ef0a3a539065" (UID: "341883ea-c8c4-49dd-ba87-ef0a3a539065"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:39.099967 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.099780 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5"] Nov 02 18:11:39.100336 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.100047 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" podUID="24f05804-b83e-44f0-b01d-b1f34904fba2" containerName="istio-proxy" containerID="cri-o://8786f64d678c887c83efbb605697260c7d8a7e437393df396e110efb1d948c58" gracePeriod=30 Nov 02 18:11:39.101394 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.101272 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/341883ea-c8c4-49dd-ba87-ef0a3a539065-istio-podinfo" pod "341883ea-c8c4-49dd-ba87-ef0a3a539065" (UID: "341883ea-c8c4-49dd-ba87-ef0a3a539065"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:11:39.106424 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.105745 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/341883ea-c8c4-49dd-ba87-ef0a3a539065-kube-api-access-229bb" pod "341883ea-c8c4-49dd-ba87-ef0a3a539065" (UID: "341883ea-c8c4-49dd-ba87-ef0a3a539065"). InnerVolumeSpecName "kube-api-access-229bb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:39.111012 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.110992 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn"] Nov 02 18:11:39.111369 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.111349 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" podUID="e881e4aa-a5ad-46d2-952d-97c5118e98b3" containerName="istio-proxy" containerID="cri-o://8563149652f5df2a3f53cb8f26fd0d8520a8dcd7d6ce35447e9870c5fcd40331" gracePeriod=30 Nov 02 18:11:39.113707 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.113687 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-istio-envoy" pod "341883ea-c8c4-49dd-ba87-ef0a3a539065" (UID: "341883ea-c8c4-49dd-ba87-ef0a3a539065"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:39.122184 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.122157 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/341883ea-c8c4-49dd-ba87-ef0a3a539065-istio-token" pod "341883ea-c8c4-49dd-ba87-ef0a3a539065" (UID: "341883ea-c8c4-49dd-ba87-ef0a3a539065"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:39.133462 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.133432 4471 generic.go:356] "Generic (PLEG): container finished" podID="341883ea-c8c4-49dd-ba87-ef0a3a539065" containerID="223b0b7f02faec9930397bf9dd171da4d453cca519329ca75cff4e1ce02d36de" exitCode=0 Nov 02 18:11:39.133614 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.133475 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" event={"ID":"341883ea-c8c4-49dd-ba87-ef0a3a539065","Type":"ContainerDied","Data":"223b0b7f02faec9930397bf9dd171da4d453cca519329ca75cff4e1ce02d36de"} Nov 02 18:11:39.133614 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.133522 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" event={"ID":"341883ea-c8c4-49dd-ba87-ef0a3a539065","Type":"ContainerDied","Data":"81507ef48369f20303ddbd18684e8c9db472e78bbb4c8d9de7f19e773794e31b"} Nov 02 18:11:39.133614 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.133554 4471 scope.go:122] "RemoveContainer" containerID="223b0b7f02faec9930397bf9dd171da4d453cca519329ca75cff4e1ce02d36de" Nov 02 18:11:39.133845 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.133726 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b" Nov 02 18:11:39.161108 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.161062 4471 scope.go:122] "RemoveContainer" containerID="b06144af5df7a10b4a043b8b6f8f86b2ab596e60b7a9142453f0d270f511fc67" Nov 02 18:11:39.211388 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.210492 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:39.211388 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.210528 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:39.211388 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.210542 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:39.211388 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.210558 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/341883ea-c8c4-49dd-ba87-ef0a3a539065-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:39.211388 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.210571 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/341883ea-c8c4-49dd-ba87-ef0a3a539065-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:39.211388 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.210583 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/341883ea-c8c4-49dd-ba87-ef0a3a539065-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:39.211388 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.210598 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-229bb\" (UniqueName: \"kubernetes.io/projected/341883ea-c8c4-49dd-ba87-ef0a3a539065-kube-api-access-229bb\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:39.279609 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.277797 4471 scope.go:122] "RemoveContainer" containerID="223b0b7f02faec9930397bf9dd171da4d453cca519329ca75cff4e1ce02d36de" Nov 02 18:11:39.283549 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:11:39.280526 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"223b0b7f02faec9930397bf9dd171da4d453cca519329ca75cff4e1ce02d36de\": container with ID starting with 223b0b7f02faec9930397bf9dd171da4d453cca519329ca75cff4e1ce02d36de not found: ID does not exist" containerID="223b0b7f02faec9930397bf9dd171da4d453cca519329ca75cff4e1ce02d36de" Nov 02 18:11:39.283549 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.280576 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"223b0b7f02faec9930397bf9dd171da4d453cca519329ca75cff4e1ce02d36de"} err="failed to get container status \"223b0b7f02faec9930397bf9dd171da4d453cca519329ca75cff4e1ce02d36de\": rpc error: code = NotFound desc = could not find container \"223b0b7f02faec9930397bf9dd171da4d453cca519329ca75cff4e1ce02d36de\": container with ID starting with 223b0b7f02faec9930397bf9dd171da4d453cca519329ca75cff4e1ce02d36de not found: ID does not exist" Nov 02 18:11:39.283549 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.280599 4471 scope.go:122] "RemoveContainer" containerID="b06144af5df7a10b4a043b8b6f8f86b2ab596e60b7a9142453f0d270f511fc67" Nov 02 18:11:39.307399 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:11:39.295249 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b06144af5df7a10b4a043b8b6f8f86b2ab596e60b7a9142453f0d270f511fc67\": container with ID starting with b06144af5df7a10b4a043b8b6f8f86b2ab596e60b7a9142453f0d270f511fc67 not found: ID does not exist" containerID="b06144af5df7a10b4a043b8b6f8f86b2ab596e60b7a9142453f0d270f511fc67" Nov 02 18:11:39.307399 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.295394 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b06144af5df7a10b4a043b8b6f8f86b2ab596e60b7a9142453f0d270f511fc67"} err="failed to get container status \"b06144af5df7a10b4a043b8b6f8f86b2ab596e60b7a9142453f0d270f511fc67\": rpc error: code = NotFound desc = could not find container \"b06144af5df7a10b4a043b8b6f8f86b2ab596e60b7a9142453f0d270f511fc67\": container with ID starting with b06144af5df7a10b4a043b8b6f8f86b2ab596e60b7a9142453f0d270f511fc67 not found: ID does not exist" Nov 02 18:11:39.408331 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.402018 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b"] Nov 02 18:11:39.409533 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.409486 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2f6787-openshift-default-77bf6f6f5f-6gk4b"] Nov 02 18:11:39.472612 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:11:39.472535 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:11:39.751817 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.751707 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2"] Nov 02 18:11:39.752001 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.751966 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" podUID="e8eeabf7-f670-46c2-bb59-e07e1a1f1337" containerName="istio-proxy" containerID="cri-o://7cbcecb8519c03d1a8274e21d4df7ffdafc32accf5c8e9e3bf46e9114ecf7652" gracePeriod=30 Nov 02 18:11:39.769965 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.769922 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq"] Nov 02 18:11:39.770165 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.770143 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" podUID="1ac349d9-ff43-493a-8cd2-3bfa0836a8da" containerName="istio-proxy" containerID="cri-o://2aef73ac8535c89039ffbaca42fd20b130d3319fc89de04458dad0d6e1c1172e" gracePeriod=30 Nov 02 18:11:39.910329 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.901053 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz"] Nov 02 18:11:39.910329 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:39.901363 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" podUID="ef139666-59eb-4847-9f3d-8fc0d20529dc" containerName="istio-proxy" containerID="cri-o://63daf3d4ceb7310bf60553eb303ae9de8030019b0e38713356b0a95fe2bfc420" gracePeriod=30 Nov 02 18:11:40.067530 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:40.067492 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv"] Nov 02 18:11:40.067761 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:40.067733 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" podUID="8210f11c-5046-4e0b-a3ca-d859918e76c7" containerName="istio-proxy" containerID="cri-o://6fb440c60b1ae9dabe51a4b1c3170b9d9f39b060fa910f0af37932cb43dacdd8" gracePeriod=30 Nov 02 18:11:40.074683 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:40.074652 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5"] Nov 02 18:11:40.074924 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:40.074899 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" podUID="6ff6c042-8820-4706-bef8-e4cf2732a389" containerName="istio-proxy" containerID="cri-o://7ffd35bc256878511d42c8959e6a6f834cd9d83ff47ab3a3ff635817057724ae" gracePeriod=30 Nov 02 18:11:40.443748 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:40.443661 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="341883ea-c8c4-49dd-ba87-ef0a3a539065" path="/var/lib/kubelet/pods/341883ea-c8c4-49dd-ba87-ef0a3a539065/volumes" Nov 02 18:11:41.226970 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:41.226932 4471 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:11:41.227482 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:41.227014 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" podUID="24f05804-b83e-44f0-b01d-b1f34904fba2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:11:42.228573 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:42.228536 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p"] Nov 02 18:11:42.229339 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:42.229286 4471 cpu_manager.go:414] "containerMap: removing container" podUID="341883ea-c8c4-49dd-ba87-ef0a3a539065" containerName="istio-proxy" Nov 02 18:11:42.229339 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:42.229316 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="341883ea-c8c4-49dd-ba87-ef0a3a539065" containerName="istio-proxy" Nov 02 18:11:42.229477 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:42.229467 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="341883ea-c8c4-49dd-ba87-ef0a3a539065" containerName="istio-proxy" Nov 02 18:11:42.233672 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:42.233641 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p" Nov 02 18:11:42.254871 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:42.254823 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-home\") pod \"auth-test-model-f91b2d-kserve-574c85fd65-tdn8p\" (UID: \"cd1e4dbc-50ca-4729-9be9-cc3f6e928a48\") " pod="ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p" Nov 02 18:11:42.255074 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:42.254924 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-tmp-dir\") pod \"auth-test-model-f91b2d-kserve-574c85fd65-tdn8p\" (UID: \"cd1e4dbc-50ca-4729-9be9-cc3f6e928a48\") " pod="ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p" Nov 02 18:11:42.255074 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:42.255064 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nlr7\" (UniqueName: \"kubernetes.io/projected/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-kube-api-access-8nlr7\") pod \"auth-test-model-f91b2d-kserve-574c85fd65-tdn8p\" (UID: \"cd1e4dbc-50ca-4729-9be9-cc3f6e928a48\") " pod="ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p" Nov 02 18:11:42.255215 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:42.255139 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-dshm\") pod \"auth-test-model-f91b2d-kserve-574c85fd65-tdn8p\" (UID: \"cd1e4dbc-50ca-4729-9be9-cc3f6e928a48\") " pod="ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p" Nov 02 18:11:42.255274 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:42.255233 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-tls-certs\") pod \"auth-test-model-f91b2d-kserve-574c85fd65-tdn8p\" (UID: \"cd1e4dbc-50ca-4729-9be9-cc3f6e928a48\") " pod="ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p" Nov 02 18:11:42.255399 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:42.255275 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-model-cache\") pod \"auth-test-model-f91b2d-kserve-574c85fd65-tdn8p\" (UID: \"cd1e4dbc-50ca-4729-9be9-cc3f6e928a48\") " pod="ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p" Nov 02 18:11:42.258083 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:42.258054 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p"] Nov 02 18:11:42.263462 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:42.263437 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-f91b2d\"/\"default-dockercfg-srptt\"" Nov 02 18:11:42.263618 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:42.263597 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-f91b2d\"/\"kube-root-ca.crt\"" Nov 02 18:11:42.263707 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:42.263602 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-f91b2d\"/\"auth-test-model-f91b2d-kserve-self-signed-certs\"" Nov 02 18:11:42.263768 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:42.263710 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-f91b2d\"/\"openshift-service-ca.crt\"" Nov 02 18:11:42.356395 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:42.356361 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-home\") pod \"auth-test-model-f91b2d-kserve-574c85fd65-tdn8p\" (UID: \"cd1e4dbc-50ca-4729-9be9-cc3f6e928a48\") " pod="ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p" Nov 02 18:11:42.356569 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:42.356432 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-tmp-dir\") pod \"auth-test-model-f91b2d-kserve-574c85fd65-tdn8p\" (UID: \"cd1e4dbc-50ca-4729-9be9-cc3f6e928a48\") " pod="ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p" Nov 02 18:11:42.356569 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:42.356492 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nlr7\" (UniqueName: \"kubernetes.io/projected/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-kube-api-access-8nlr7\") pod \"auth-test-model-f91b2d-kserve-574c85fd65-tdn8p\" (UID: \"cd1e4dbc-50ca-4729-9be9-cc3f6e928a48\") " pod="ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p" Nov 02 18:11:42.356569 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:42.356537 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-dshm\") pod \"auth-test-model-f91b2d-kserve-574c85fd65-tdn8p\" (UID: \"cd1e4dbc-50ca-4729-9be9-cc3f6e928a48\") " pod="ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p" Nov 02 18:11:42.356719 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:42.356614 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-tls-certs\") pod \"auth-test-model-f91b2d-kserve-574c85fd65-tdn8p\" (UID: \"cd1e4dbc-50ca-4729-9be9-cc3f6e928a48\") " pod="ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p" Nov 02 18:11:42.356719 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:42.356652 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-model-cache\") pod \"auth-test-model-f91b2d-kserve-574c85fd65-tdn8p\" (UID: \"cd1e4dbc-50ca-4729-9be9-cc3f6e928a48\") " pod="ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p" Nov 02 18:11:42.356832 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:42.356777 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-home\") pod \"auth-test-model-f91b2d-kserve-574c85fd65-tdn8p\" (UID: \"cd1e4dbc-50ca-4729-9be9-cc3f6e928a48\") " pod="ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p" Nov 02 18:11:42.356832 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:42.356807 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-tmp-dir\") pod \"auth-test-model-f91b2d-kserve-574c85fd65-tdn8p\" (UID: \"cd1e4dbc-50ca-4729-9be9-cc3f6e928a48\") " pod="ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p" Nov 02 18:11:42.357071 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:42.357050 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-model-cache\") pod \"auth-test-model-f91b2d-kserve-574c85fd65-tdn8p\" (UID: \"cd1e4dbc-50ca-4729-9be9-cc3f6e928a48\") " pod="ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p" Nov 02 18:11:42.358832 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:42.358808 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-dshm\") pod \"auth-test-model-f91b2d-kserve-574c85fd65-tdn8p\" (UID: \"cd1e4dbc-50ca-4729-9be9-cc3f6e928a48\") " pod="ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p" Nov 02 18:11:42.359113 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:42.359097 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-tls-certs\") pod \"auth-test-model-f91b2d-kserve-574c85fd65-tdn8p\" (UID: \"cd1e4dbc-50ca-4729-9be9-cc3f6e928a48\") " pod="ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p" Nov 02 18:11:42.380787 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:42.380759 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nlr7\" (UniqueName: \"kubernetes.io/projected/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-kube-api-access-8nlr7\") pod \"auth-test-model-f91b2d-kserve-574c85fd65-tdn8p\" (UID: \"cd1e4dbc-50ca-4729-9be9-cc3f6e928a48\") " pod="ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p" Nov 02 18:11:42.425733 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:42.425697 4471 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:11:42.425896 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:42.425765 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" podUID="e881e4aa-a5ad-46d2-952d-97c5118e98b3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:11:42.435857 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:42.435820 4471 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:11:42.436032 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:42.435913 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" podUID="38fea887-fcc0-42fa-a1c9-320e44f70562" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:11:42.557336 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:42.556656 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p" Nov 02 18:11:42.887919 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:11:42.887887 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd1e4dbc_50ca_4729_9be9_cc3f6e928a48.slice/crio-04db58c689a1b469ec9e168e2e4c3605824462209d713963bfd21af13e0d692c WatchSource:0}: Error finding container 04db58c689a1b469ec9e168e2e4c3605824462209d713963bfd21af13e0d692c: Status 404 returned error can't find the container with id 04db58c689a1b469ec9e168e2e4c3605824462209d713963bfd21af13e0d692c Nov 02 18:11:42.891136 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:42.891108 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p"] Nov 02 18:11:43.169265 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:43.169177 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p" event={"ID":"cd1e4dbc-50ca-4729-9be9-cc3f6e928a48","Type":"ContainerStarted","Data":"538312ab8c10efac3f699eecbe58273506314ccab6c5484869f154c0be76c7de"} Nov 02 18:11:43.169265 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:43.169229 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p" event={"ID":"cd1e4dbc-50ca-4729-9be9-cc3f6e928a48","Type":"ContainerStarted","Data":"04db58c689a1b469ec9e168e2e4c3605824462209d713963bfd21af13e0d692c"} Nov 02 18:11:43.557443 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:43.557407 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p" Nov 02 18:11:44.422181 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.422153 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 18:11:44.496844 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.496790 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p" podStartSLOduration=2.496775791 podStartE2EDuration="2.496775791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:11:43.22361333 +0000 UTC m=+2520.918968805" watchObservedRunningTime="2026-11-02 18:11:44.496775791 +0000 UTC m=+2522.192131541" Nov 02 18:11:44.511351 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.511324 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 18:11:44.516685 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.516650 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 18:11:44.532777 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.532746 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24f05804-b83e-44f0-b01d-b1f34904fba2-istio-token\" (UniqueName: \"kubernetes.io/projected/24f05804-b83e-44f0-b01d-b1f34904fba2-istio-token\") pod \"24f05804-b83e-44f0-b01d-b1f34904fba2\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " Nov 02 18:11:44.532917 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.532838 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-workload-certs\") pod \"24f05804-b83e-44f0-b01d-b1f34904fba2\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " Nov 02 18:11:44.532917 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.532882 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-istio-envoy\") pod \"24f05804-b83e-44f0-b01d-b1f34904fba2\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " Nov 02 18:11:44.533025 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.532959 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38fea887-fcc0-42fa-a1c9-320e44f70562-istio-token\" (UniqueName: \"kubernetes.io/projected/38fea887-fcc0-42fa-a1c9-320e44f70562-istio-token\") pod \"38fea887-fcc0-42fa-a1c9-320e44f70562\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " Nov 02 18:11:44.533025 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.532991 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24f05804-b83e-44f0-b01d-b1f34904fba2-kube-api-access-gb7qk\" (UniqueName: \"kubernetes.io/projected/24f05804-b83e-44f0-b01d-b1f34904fba2-kube-api-access-gb7qk\") pod \"24f05804-b83e-44f0-b01d-b1f34904fba2\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " Nov 02 18:11:44.533103 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.533051 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-istio-envoy\") pod \"38fea887-fcc0-42fa-a1c9-320e44f70562\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " Nov 02 18:11:44.533571 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.533521 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-workload-certs" pod "24f05804-b83e-44f0-b01d-b1f34904fba2" (UID: "24f05804-b83e-44f0-b01d-b1f34904fba2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:44.533720 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.533592 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38fea887-fcc0-42fa-a1c9-320e44f70562-kube-api-access-75csh\" (UniqueName: \"kubernetes.io/projected/38fea887-fcc0-42fa-a1c9-320e44f70562-kube-api-access-75csh\") pod \"38fea887-fcc0-42fa-a1c9-320e44f70562\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " Nov 02 18:11:44.533720 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.533712 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-workload-socket\") pod \"24f05804-b83e-44f0-b01d-b1f34904fba2\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " Nov 02 18:11:44.535979 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.534065 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38fea887-fcc0-42fa-a1c9-320e44f70562-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38fea887-fcc0-42fa-a1c9-320e44f70562-istiod-ca-cert\") pod \"38fea887-fcc0-42fa-a1c9-320e44f70562\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " Nov 02 18:11:44.535979 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.534105 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38fea887-fcc0-42fa-a1c9-320e44f70562-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38fea887-fcc0-42fa-a1c9-320e44f70562-istio-podinfo\") pod \"38fea887-fcc0-42fa-a1c9-320e44f70562\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " Nov 02 18:11:44.535979 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.534187 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-workload-socket" pod "24f05804-b83e-44f0-b01d-b1f34904fba2" (UID: "24f05804-b83e-44f0-b01d-b1f34904fba2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:44.535979 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.534241 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-istio-data\") pod \"38fea887-fcc0-42fa-a1c9-320e44f70562\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " Nov 02 18:11:44.535979 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.534334 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-workload-certs\") pod \"38fea887-fcc0-42fa-a1c9-320e44f70562\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " Nov 02 18:11:44.535979 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.534391 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-credential-socket\") pod \"38fea887-fcc0-42fa-a1c9-320e44f70562\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " Nov 02 18:11:44.535979 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.534444 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-istio-data\") pod \"24f05804-b83e-44f0-b01d-b1f34904fba2\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " Nov 02 18:11:44.535979 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.534475 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38fea887-fcc0-42fa-a1c9-320e44f70562-istiod-ca-cert" pod "38fea887-fcc0-42fa-a1c9-320e44f70562" (UID: "38fea887-fcc0-42fa-a1c9-320e44f70562"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:44.535979 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.534485 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-credential-socket\") pod \"24f05804-b83e-44f0-b01d-b1f34904fba2\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " Nov 02 18:11:44.535979 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.534548 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-workload-socket\") pod \"38fea887-fcc0-42fa-a1c9-320e44f70562\" (UID: \"38fea887-fcc0-42fa-a1c9-320e44f70562\") " Nov 02 18:11:44.535979 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.534703 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-credential-socket" pod "24f05804-b83e-44f0-b01d-b1f34904fba2" (UID: "24f05804-b83e-44f0-b01d-b1f34904fba2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:44.535979 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.534906 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-credential-socket" pod "38fea887-fcc0-42fa-a1c9-320e44f70562" (UID: "38fea887-fcc0-42fa-a1c9-320e44f70562"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:44.535979 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.535057 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-istio-data" pod "24f05804-b83e-44f0-b01d-b1f34904fba2" (UID: "24f05804-b83e-44f0-b01d-b1f34904fba2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:44.535979 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.535114 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-istio-data" pod "38fea887-fcc0-42fa-a1c9-320e44f70562" (UID: "38fea887-fcc0-42fa-a1c9-320e44f70562"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:44.535979 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.535179 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-workload-certs" pod "38fea887-fcc0-42fa-a1c9-320e44f70562" (UID: "38fea887-fcc0-42fa-a1c9-320e44f70562"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:44.535979 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.535299 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24f05804-b83e-44f0-b01d-b1f34904fba2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24f05804-b83e-44f0-b01d-b1f34904fba2-istiod-ca-cert\") pod \"24f05804-b83e-44f0-b01d-b1f34904fba2\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " Nov 02 18:11:44.535979 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.535371 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-workload-socket" pod "38fea887-fcc0-42fa-a1c9-320e44f70562" (UID: "38fea887-fcc0-42fa-a1c9-320e44f70562"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:44.535979 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.535381 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24f05804-b83e-44f0-b01d-b1f34904fba2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24f05804-b83e-44f0-b01d-b1f34904fba2-istio-podinfo\") pod \"24f05804-b83e-44f0-b01d-b1f34904fba2\" (UID: \"24f05804-b83e-44f0-b01d-b1f34904fba2\") " Nov 02 18:11:44.535979 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.535675 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24f05804-b83e-44f0-b01d-b1f34904fba2-istiod-ca-cert" pod "24f05804-b83e-44f0-b01d-b1f34904fba2" (UID: "24f05804-b83e-44f0-b01d-b1f34904fba2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:44.537079 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.536379 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.537079 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.536401 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.537079 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.536414 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.537079 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.536426 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24f05804-b83e-44f0-b01d-b1f34904fba2-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.537079 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.536441 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.537079 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.536454 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.537079 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.536467 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38fea887-fcc0-42fa-a1c9-320e44f70562-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.537079 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.536479 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.537079 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.536490 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.537079 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.536529 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.537944 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.537497 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f05804-b83e-44f0-b01d-b1f34904fba2-istio-token" pod "24f05804-b83e-44f0-b01d-b1f34904fba2" (UID: "24f05804-b83e-44f0-b01d-b1f34904fba2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:44.537944 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.537531 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f05804-b83e-44f0-b01d-b1f34904fba2-kube-api-access-gb7qk" pod "24f05804-b83e-44f0-b01d-b1f34904fba2" (UID: "24f05804-b83e-44f0-b01d-b1f34904fba2"). InnerVolumeSpecName "kube-api-access-gb7qk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:44.537944 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.537552 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38fea887-fcc0-42fa-a1c9-320e44f70562-istio-token" pod "38fea887-fcc0-42fa-a1c9-320e44f70562" (UID: "38fea887-fcc0-42fa-a1c9-320e44f70562"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:44.537944 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.537606 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38fea887-fcc0-42fa-a1c9-320e44f70562-istio-podinfo" pod "38fea887-fcc0-42fa-a1c9-320e44f70562" (UID: "38fea887-fcc0-42fa-a1c9-320e44f70562"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:11:44.539018 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.538994 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24f05804-b83e-44f0-b01d-b1f34904fba2-istio-podinfo" pod "24f05804-b83e-44f0-b01d-b1f34904fba2" (UID: "24f05804-b83e-44f0-b01d-b1f34904fba2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:11:44.541693 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.541669 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-istio-envoy" pod "24f05804-b83e-44f0-b01d-b1f34904fba2" (UID: "24f05804-b83e-44f0-b01d-b1f34904fba2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:44.542077 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.542052 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38fea887-fcc0-42fa-a1c9-320e44f70562-kube-api-access-75csh" pod "38fea887-fcc0-42fa-a1c9-320e44f70562" (UID: "38fea887-fcc0-42fa-a1c9-320e44f70562"). InnerVolumeSpecName "kube-api-access-75csh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:44.543349 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.543289 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-istio-envoy" pod "38fea887-fcc0-42fa-a1c9-320e44f70562" (UID: "38fea887-fcc0-42fa-a1c9-320e44f70562"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:44.638187 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.638096 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-workload-socket\") pod \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " Nov 02 18:11:44.638187 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.638170 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istio-token\" (UniqueName: \"kubernetes.io/projected/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istio-token\") pod \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " Nov 02 18:11:44.638631 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.638238 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e881e4aa-a5ad-46d2-952d-97c5118e98b3-kube-api-access-nzxwm\" (UniqueName: \"kubernetes.io/projected/e881e4aa-a5ad-46d2-952d-97c5118e98b3-kube-api-access-nzxwm\") pod \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " Nov 02 18:11:44.638631 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.638301 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istio-envoy\") pod \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " Nov 02 18:11:44.638631 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.638369 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istiod-ca-cert\") pod \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " Nov 02 18:11:44.638631 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.638443 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-workload-socket" pod "e881e4aa-a5ad-46d2-952d-97c5118e98b3" (UID: "e881e4aa-a5ad-46d2-952d-97c5118e98b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:44.638631 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.638460 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-workload-certs\") pod \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " Nov 02 18:11:44.638808 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.638701 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istio-data\") pod \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " Nov 02 18:11:44.638808 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.638709 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-workload-certs" pod "e881e4aa-a5ad-46d2-952d-97c5118e98b3" (UID: "e881e4aa-a5ad-46d2-952d-97c5118e98b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:44.638951 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.638930 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istio-data" pod "e881e4aa-a5ad-46d2-952d-97c5118e98b3" (UID: "e881e4aa-a5ad-46d2-952d-97c5118e98b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:44.639167 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.639147 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istio-podinfo\") pod \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " Nov 02 18:11:44.639268 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.639255 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-credential-socket\") pod \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\" (UID: \"e881e4aa-a5ad-46d2-952d-97c5118e98b3\") " Nov 02 18:11:44.640131 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.640106 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.640255 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.640135 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24f05804-b83e-44f0-b01d-b1f34904fba2-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.640255 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.640149 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24f05804-b83e-44f0-b01d-b1f34904fba2-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.640255 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.640162 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.640255 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.640174 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24f05804-b83e-44f0-b01d-b1f34904fba2-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.640255 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.640187 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38fea887-fcc0-42fa-a1c9-320e44f70562-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.640255 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.640199 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb7qk\" (UniqueName: \"kubernetes.io/projected/24f05804-b83e-44f0-b01d-b1f34904fba2-kube-api-access-gb7qk\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.640255 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.640215 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38fea887-fcc0-42fa-a1c9-320e44f70562-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.640255 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.640227 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.640255 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.640240 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75csh\" (UniqueName: \"kubernetes.io/projected/38fea887-fcc0-42fa-a1c9-320e44f70562-kube-api-access-75csh\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.640857 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.640259 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38fea887-fcc0-42fa-a1c9-320e44f70562-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.640857 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.640494 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-credential-socket" pod "e881e4aa-a5ad-46d2-952d-97c5118e98b3" (UID: "e881e4aa-a5ad-46d2-952d-97c5118e98b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:44.641013 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.640990 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istiod-ca-cert" pod "e881e4aa-a5ad-46d2-952d-97c5118e98b3" (UID: "e881e4aa-a5ad-46d2-952d-97c5118e98b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:44.641660 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.641627 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istio-envoy" pod "e881e4aa-a5ad-46d2-952d-97c5118e98b3" (UID: "e881e4aa-a5ad-46d2-952d-97c5118e98b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:44.641796 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.641772 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e881e4aa-a5ad-46d2-952d-97c5118e98b3-kube-api-access-nzxwm" pod "e881e4aa-a5ad-46d2-952d-97c5118e98b3" (UID: "e881e4aa-a5ad-46d2-952d-97c5118e98b3"). InnerVolumeSpecName "kube-api-access-nzxwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:44.642382 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.642352 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istio-token" pod "e881e4aa-a5ad-46d2-952d-97c5118e98b3" (UID: "e881e4aa-a5ad-46d2-952d-97c5118e98b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:44.643142 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.643121 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istio-podinfo" pod "e881e4aa-a5ad-46d2-952d-97c5118e98b3" (UID: "e881e4aa-a5ad-46d2-952d-97c5118e98b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:11:44.708831 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.708780 4471 scope.go:122] "RemoveContainer" containerID="be7de622429723d40162e60b9a5cc831063246abf102b08690692d47c0252773" Nov 02 18:11:44.724192 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.724163 4471 scope.go:122] "RemoveContainer" containerID="a4448c42d1a716a6e6b50f1746bbd530359edc623fb4fd91fed9aeb3df020fa3" Nov 02 18:11:44.734747 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.734719 4471 scope.go:122] "RemoveContainer" containerID="8563149652f5df2a3f53cb8f26fd0d8520a8dcd7d6ce35447e9870c5fcd40331" Nov 02 18:11:44.742083 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.742051 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.742083 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.742081 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzxwm\" (UniqueName: \"kubernetes.io/projected/e881e4aa-a5ad-46d2-952d-97c5118e98b3-kube-api-access-nzxwm\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.742235 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.742097 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.742235 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.742111 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.742235 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.742122 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e881e4aa-a5ad-46d2-952d-97c5118e98b3-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.742235 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.742135 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e881e4aa-a5ad-46d2-952d-97c5118e98b3-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.748659 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.748508 4471 scope.go:122] "RemoveContainer" containerID="8786f64d678c887c83efbb605697260c7d8a7e437393df396e110efb1d948c58" Nov 02 18:11:44.764933 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.764905 4471 scope.go:122] "RemoveContainer" containerID="2f33607f9e421093ec91c99de5a98130292399afa5ee9d6fa12df8e0e7e23650" Nov 02 18:11:44.994285 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:44.992998 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8a209f-789b5bc59f-lgg4z"] Nov 02 18:11:45.076838 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.076805 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 18:11:45.155810 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.155729 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istiod-ca-cert\") pod \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " Nov 02 18:11:45.155948 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.155825 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-kube-api-access-jvnwl\" (UniqueName: \"kubernetes.io/projected/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-kube-api-access-jvnwl\") pod \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " Nov 02 18:11:45.155948 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.155897 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istio-envoy\") pod \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " Nov 02 18:11:45.156024 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.156007 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istio-podinfo\") pod \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " Nov 02 18:11:45.156057 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.156037 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-workload-certs\") pod \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " Nov 02 18:11:45.156116 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.156098 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-workload-socket\") pod \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " Nov 02 18:11:45.156200 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.156186 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istio-data\") pod \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " Nov 02 18:11:45.156286 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.156269 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-credential-socket\") pod \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " Nov 02 18:11:45.156455 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.156436 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istio-token\" (UniqueName: \"kubernetes.io/projected/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istio-token\") pod \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\" (UID: \"e8eeabf7-f670-46c2-bb59-e07e1a1f1337\") " Nov 02 18:11:45.156675 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.156646 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-workload-socket" pod "e8eeabf7-f670-46c2-bb59-e07e1a1f1337" (UID: "e8eeabf7-f670-46c2-bb59-e07e1a1f1337"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.156918 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.156893 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-workload-certs" pod "e8eeabf7-f670-46c2-bb59-e07e1a1f1337" (UID: "e8eeabf7-f670-46c2-bb59-e07e1a1f1337"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.157284 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.157255 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.157399 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.157289 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.157399 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.157332 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istiod-ca-cert" pod "e8eeabf7-f670-46c2-bb59-e07e1a1f1337" (UID: "e8eeabf7-f670-46c2-bb59-e07e1a1f1337"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:45.157587 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.157397 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-credential-socket" pod "e8eeabf7-f670-46c2-bb59-e07e1a1f1337" (UID: "e8eeabf7-f670-46c2-bb59-e07e1a1f1337"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.157587 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.157450 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istio-data" pod "e8eeabf7-f670-46c2-bb59-e07e1a1f1337" (UID: "e8eeabf7-f670-46c2-bb59-e07e1a1f1337"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.159600 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.159578 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istio-podinfo" pod "e8eeabf7-f670-46c2-bb59-e07e1a1f1337" (UID: "e8eeabf7-f670-46c2-bb59-e07e1a1f1337"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:11:45.159768 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.159751 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-kube-api-access-jvnwl" pod "e8eeabf7-f670-46c2-bb59-e07e1a1f1337" (UID: "e8eeabf7-f670-46c2-bb59-e07e1a1f1337"). InnerVolumeSpecName "kube-api-access-jvnwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:45.159935 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.159910 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istio-token" pod "e8eeabf7-f670-46c2-bb59-e07e1a1f1337" (UID: "e8eeabf7-f670-46c2-bb59-e07e1a1f1337"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:45.161404 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.161382 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istio-envoy" pod "e8eeabf7-f670-46c2-bb59-e07e1a1f1337" (UID: "e8eeabf7-f670-46c2-bb59-e07e1a1f1337"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.258697 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.258657 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvnwl\" (UniqueName: \"kubernetes.io/projected/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-kube-api-access-jvnwl\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.258697 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.258696 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.258955 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.258712 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.258955 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.258725 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.258955 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.258737 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.258955 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.258750 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.258955 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.258763 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8eeabf7-f670-46c2-bb59-e07e1a1f1337-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.296095 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.295978 4471 generic.go:356] "Generic (PLEG): container finished" podID="1ac349d9-ff43-493a-8cd2-3bfa0836a8da" containerID="2aef73ac8535c89039ffbaca42fd20b130d3319fc89de04458dad0d6e1c1172e" exitCode=0 Nov 02 18:11:45.298760 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.298733 4471 generic.go:356] "Generic (PLEG): container finished" podID="e8eeabf7-f670-46c2-bb59-e07e1a1f1337" containerID="7cbcecb8519c03d1a8274e21d4df7ffdafc32accf5c8e9e3bf46e9114ecf7652" exitCode=0 Nov 02 18:11:45.298938 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.298867 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" Nov 02 18:11:45.301625 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.301545 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 18:11:45.303623 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.303591 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" event={"ID":"1ac349d9-ff43-493a-8cd2-3bfa0836a8da","Type":"ContainerDied","Data":"2aef73ac8535c89039ffbaca42fd20b130d3319fc89de04458dad0d6e1c1172e"} Nov 02 18:11:45.304450 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.304425 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" event={"ID":"e8eeabf7-f670-46c2-bb59-e07e1a1f1337","Type":"ContainerDied","Data":"7cbcecb8519c03d1a8274e21d4df7ffdafc32accf5c8e9e3bf46e9114ecf7652"} Nov 02 18:11:45.305732 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.304603 4471 scope.go:122] "RemoveContainer" containerID="2aef73ac8535c89039ffbaca42fd20b130d3319fc89de04458dad0d6e1c1172e" Nov 02 18:11:45.305732 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.305494 4471 generic.go:356] "Generic (PLEG): container finished" podID="ef139666-59eb-4847-9f3d-8fc0d20529dc" containerID="63daf3d4ceb7310bf60553eb303ae9de8030019b0e38713356b0a95fe2bfc420" exitCode=0 Nov 02 18:11:45.309024 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.308999 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2" event={"ID":"e8eeabf7-f670-46c2-bb59-e07e1a1f1337","Type":"ContainerDied","Data":"6e554d9c682c3fb5296620d8dd51afa4a09b7d58eef694da70fd7b9de027b717"} Nov 02 18:11:45.309194 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.309179 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" event={"ID":"ef139666-59eb-4847-9f3d-8fc0d20529dc","Type":"ContainerDied","Data":"63daf3d4ceb7310bf60553eb303ae9de8030019b0e38713356b0a95fe2bfc420"} Nov 02 18:11:45.310147 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.310125 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" event={"ID":"38fea887-fcc0-42fa-a1c9-320e44f70562","Type":"ContainerDied","Data":"2f33607f9e421093ec91c99de5a98130292399afa5ee9d6fa12df8e0e7e23650"} Nov 02 18:11:45.311842 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.311733 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" event={"ID":"38fea887-fcc0-42fa-a1c9-320e44f70562","Type":"ContainerDied","Data":"11a5437f8ec75661c50f6b6373cad7035321997d4f1e1e964da881b43434d8b0"} Nov 02 18:11:45.311842 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.311775 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" event={"ID":"24f05804-b83e-44f0-b01d-b1f34904fba2","Type":"ContainerDied","Data":"8786f64d678c887c83efbb605697260c7d8a7e437393df396e110efb1d948c58"} Nov 02 18:11:45.311842 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.311799 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" event={"ID":"24f05804-b83e-44f0-b01d-b1f34904fba2","Type":"ContainerDied","Data":"8ae25c8c6b88e8f87c6c6118a99caa7b1cd05343f08bfaa2cafaf2d28323c5b1"} Nov 02 18:11:45.312044 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.311896 4471 generic.go:356] "Generic (PLEG): container finished" podID="8210f11c-5046-4e0b-a3ca-d859918e76c7" containerID="6fb440c60b1ae9dabe51a4b1c3170b9d9f39b060fa910f0af37932cb43dacdd8" exitCode=0 Nov 02 18:11:45.312044 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.311954 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" event={"ID":"8210f11c-5046-4e0b-a3ca-d859918e76c7","Type":"ContainerDied","Data":"6fb440c60b1ae9dabe51a4b1c3170b9d9f39b060fa910f0af37932cb43dacdd8"} Nov 02 18:11:45.312141 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.312097 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5" Nov 02 18:11:45.312141 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.312123 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" event={"ID":"e881e4aa-a5ad-46d2-952d-97c5118e98b3","Type":"ContainerDied","Data":"8563149652f5df2a3f53cb8f26fd0d8520a8dcd7d6ce35447e9870c5fcd40331"} Nov 02 18:11:45.313043 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.312187 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" event={"ID":"e881e4aa-a5ad-46d2-952d-97c5118e98b3","Type":"ContainerDied","Data":"f2d23fa64798b2c0c58eb3c4a6b293d13c37f438ed2462d2d3bd111968822bda"} Nov 02 18:11:45.313043 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.312401 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn" Nov 02 18:11:45.318048 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.318015 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9" Nov 02 18:11:45.333391 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.333364 4471 scope.go:122] "RemoveContainer" containerID="7cbcecb8519c03d1a8274e21d4df7ffdafc32accf5c8e9e3bf46e9114ecf7652" Nov 02 18:11:45.359538 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.359503 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-credential-socket\") pod \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " Nov 02 18:11:45.359672 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.359630 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istio-envoy\") pod \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " Nov 02 18:11:45.359727 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.359695 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istiod-ca-cert\") pod \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " Nov 02 18:11:45.359774 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.359745 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istio-podinfo\") pod \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " Nov 02 18:11:45.359901 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.359884 4471 scope.go:122] "RemoveContainer" containerID="7cbcecb8519c03d1a8274e21d4df7ffdafc32accf5c8e9e3bf46e9114ecf7652" Nov 02 18:11:45.360006 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.359901 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-workload-certs\") pod \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " Nov 02 18:11:45.360462 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.360437 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istio-data\") pod \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " Nov 02 18:11:45.360663 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.360645 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-workload-socket\") pod \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " Nov 02 18:11:45.360837 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.360819 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istio-token\" (UniqueName: \"kubernetes.io/projected/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istio-token\") pod \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " Nov 02 18:11:45.360981 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.360964 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-kube-api-access-cfxls\" (UniqueName: \"kubernetes.io/projected/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-kube-api-access-cfxls\") pod \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\" (UID: \"1ac349d9-ff43-493a-8cd2-3bfa0836a8da\") " Nov 02 18:11:45.361108 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.361084 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-credential-socket" pod "1ac349d9-ff43-493a-8cd2-3bfa0836a8da" (UID: "1ac349d9-ff43-493a-8cd2-3bfa0836a8da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.361408 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.361279 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-workload-socket" pod "1ac349d9-ff43-493a-8cd2-3bfa0836a8da" (UID: "1ac349d9-ff43-493a-8cd2-3bfa0836a8da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.361508 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.361478 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istio-data" pod "1ac349d9-ff43-493a-8cd2-3bfa0836a8da" (UID: "1ac349d9-ff43-493a-8cd2-3bfa0836a8da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.361790 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.361767 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-workload-certs" pod "1ac349d9-ff43-493a-8cd2-3bfa0836a8da" (UID: "1ac349d9-ff43-493a-8cd2-3bfa0836a8da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.361947 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.361928 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istiod-ca-cert" pod "1ac349d9-ff43-493a-8cd2-3bfa0836a8da" (UID: "1ac349d9-ff43-493a-8cd2-3bfa0836a8da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:45.362656 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.362630 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.362800 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.362783 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.362914 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.362902 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.363012 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.363000 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.363107 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.363095 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.363186 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:11:45.363047 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cbcecb8519c03d1a8274e21d4df7ffdafc32accf5c8e9e3bf46e9114ecf7652\": container with ID starting with 7cbcecb8519c03d1a8274e21d4df7ffdafc32accf5c8e9e3bf46e9114ecf7652 not found: ID does not exist" containerID="7cbcecb8519c03d1a8274e21d4df7ffdafc32accf5c8e9e3bf46e9114ecf7652" Nov 02 18:11:45.363324 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.363278 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cbcecb8519c03d1a8274e21d4df7ffdafc32accf5c8e9e3bf46e9114ecf7652"} err="failed to get container status \"7cbcecb8519c03d1a8274e21d4df7ffdafc32accf5c8e9e3bf46e9114ecf7652\": rpc error: code = NotFound desc = could not find container \"7cbcecb8519c03d1a8274e21d4df7ffdafc32accf5c8e9e3bf46e9114ecf7652\": container with ID starting with 7cbcecb8519c03d1a8274e21d4df7ffdafc32accf5c8e9e3bf46e9114ecf7652 not found: ID does not exist" Nov 02 18:11:45.365008 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.364989 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-kube-api-access-cfxls" pod "1ac349d9-ff43-493a-8cd2-3bfa0836a8da" (UID: "1ac349d9-ff43-493a-8cd2-3bfa0836a8da"). InnerVolumeSpecName "kube-api-access-cfxls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:45.379407 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.379281 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istio-podinfo" pod "1ac349d9-ff43-493a-8cd2-3bfa0836a8da" (UID: "1ac349d9-ff43-493a-8cd2-3bfa0836a8da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:11:45.382728 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.382667 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istio-token" pod "1ac349d9-ff43-493a-8cd2-3bfa0836a8da" (UID: "1ac349d9-ff43-493a-8cd2-3bfa0836a8da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:45.394514 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.394457 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istio-envoy" pod "1ac349d9-ff43-493a-8cd2-3bfa0836a8da" (UID: "1ac349d9-ff43-493a-8cd2-3bfa0836a8da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.425177 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.425087 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx"] Nov 02 18:11:45.470257 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.468004 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.470257 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.468048 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.470257 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.468063 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfxls\" (UniqueName: \"kubernetes.io/projected/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-kube-api-access-cfxls\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.470257 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.468079 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ac349d9-ff43-493a-8cd2-3bfa0836a8da-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.531477 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.531402 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5"] Nov 02 18:11:45.536803 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.536764 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-ldxg5"] Nov 02 18:11:45.565410 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.565378 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:11:45.608586 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.608543 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9"] Nov 02 18:11:45.614401 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.614370 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2bgr9"] Nov 02 18:11:45.619552 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.619497 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:11:45.649772 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.649715 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2"] Nov 02 18:11:45.653738 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.653703 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-k5qp2"] Nov 02 18:11:45.670237 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.670201 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8210f11c-5046-4e0b-a3ca-d859918e76c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8210f11c-5046-4e0b-a3ca-d859918e76c7-istio-podinfo\") pod \"8210f11c-5046-4e0b-a3ca-d859918e76c7\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " Nov 02 18:11:45.670500 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.670470 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8210f11c-5046-4e0b-a3ca-d859918e76c7-kube-api-access-x4hxm\" (UniqueName: \"kubernetes.io/projected/8210f11c-5046-4e0b-a3ca-d859918e76c7-kube-api-access-x4hxm\") pod \"8210f11c-5046-4e0b-a3ca-d859918e76c7\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " Nov 02 18:11:45.672279 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.670756 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-istio-envoy\") pod \"8210f11c-5046-4e0b-a3ca-d859918e76c7\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " Nov 02 18:11:45.672279 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.670828 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ff6c042-8820-4706-bef8-e4cf2732a389-istio-token\" (UniqueName: \"kubernetes.io/projected/6ff6c042-8820-4706-bef8-e4cf2732a389-istio-token\") pod \"6ff6c042-8820-4706-bef8-e4cf2732a389\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " Nov 02 18:11:45.672279 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.670893 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-credential-socket\") pod \"6ff6c042-8820-4706-bef8-e4cf2732a389\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " Nov 02 18:11:45.672279 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.670933 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-workload-certs\") pod \"8210f11c-5046-4e0b-a3ca-d859918e76c7\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " Nov 02 18:11:45.672279 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.670959 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-workload-socket\") pod \"6ff6c042-8820-4706-bef8-e4cf2732a389\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " Nov 02 18:11:45.672279 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.670989 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-istio-envoy\") pod \"6ff6c042-8820-4706-bef8-e4cf2732a389\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " Nov 02 18:11:45.672279 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.671032 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-istio-data\") pod \"8210f11c-5046-4e0b-a3ca-d859918e76c7\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " Nov 02 18:11:45.672279 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.671132 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-istio-data\") pod \"6ff6c042-8820-4706-bef8-e4cf2732a389\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " Nov 02 18:11:45.672279 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.671166 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ff6c042-8820-4706-bef8-e4cf2732a389-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff6c042-8820-4706-bef8-e4cf2732a389-istio-podinfo\") pod \"6ff6c042-8820-4706-bef8-e4cf2732a389\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " Nov 02 18:11:45.672279 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.671243 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8210f11c-5046-4e0b-a3ca-d859918e76c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8210f11c-5046-4e0b-a3ca-d859918e76c7-istiod-ca-cert\") pod \"8210f11c-5046-4e0b-a3ca-d859918e76c7\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " Nov 02 18:11:45.672279 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.671283 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ff6c042-8820-4706-bef8-e4cf2732a389-kube-api-access-hc4x9\" (UniqueName: \"kubernetes.io/projected/6ff6c042-8820-4706-bef8-e4cf2732a389-kube-api-access-hc4x9\") pod \"6ff6c042-8820-4706-bef8-e4cf2732a389\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " Nov 02 18:11:45.672279 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.671417 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-workload-certs\") pod \"6ff6c042-8820-4706-bef8-e4cf2732a389\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " Nov 02 18:11:45.672279 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.671447 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8210f11c-5046-4e0b-a3ca-d859918e76c7-istio-token\" (UniqueName: \"kubernetes.io/projected/8210f11c-5046-4e0b-a3ca-d859918e76c7-istio-token\") pod \"8210f11c-5046-4e0b-a3ca-d859918e76c7\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " Nov 02 18:11:45.672279 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.671546 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-credential-socket\") pod \"8210f11c-5046-4e0b-a3ca-d859918e76c7\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " Nov 02 18:11:45.672279 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.671599 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ff6c042-8820-4706-bef8-e4cf2732a389-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff6c042-8820-4706-bef8-e4cf2732a389-istiod-ca-cert\") pod \"6ff6c042-8820-4706-bef8-e4cf2732a389\" (UID: \"6ff6c042-8820-4706-bef8-e4cf2732a389\") " Nov 02 18:11:45.672279 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.671647 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-workload-socket\") pod \"8210f11c-5046-4e0b-a3ca-d859918e76c7\" (UID: \"8210f11c-5046-4e0b-a3ca-d859918e76c7\") " Nov 02 18:11:45.672279 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.672075 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-istio-data" pod "8210f11c-5046-4e0b-a3ca-d859918e76c7" (UID: "8210f11c-5046-4e0b-a3ca-d859918e76c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.673498 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.672298 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-workload-socket" pod "6ff6c042-8820-4706-bef8-e4cf2732a389" (UID: "6ff6c042-8820-4706-bef8-e4cf2732a389"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.673498 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.672543 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-credential-socket" pod "6ff6c042-8820-4706-bef8-e4cf2732a389" (UID: "6ff6c042-8820-4706-bef8-e4cf2732a389"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.674925 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.673738 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-workload-certs" pod "8210f11c-5046-4e0b-a3ca-d859918e76c7" (UID: "8210f11c-5046-4e0b-a3ca-d859918e76c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.674925 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.674168 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8210f11c-5046-4e0b-a3ca-d859918e76c7-istiod-ca-cert" pod "8210f11c-5046-4e0b-a3ca-d859918e76c7" (UID: "8210f11c-5046-4e0b-a3ca-d859918e76c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:45.674925 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.674571 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.674925 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.674594 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.674925 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.674771 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.674925 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.674789 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.674925 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.674805 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8210f11c-5046-4e0b-a3ca-d859918e76c7-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.674925 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.674614 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-workload-socket" pod "8210f11c-5046-4e0b-a3ca-d859918e76c7" (UID: "8210f11c-5046-4e0b-a3ca-d859918e76c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.674925 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.674719 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-istio-data" pod "6ff6c042-8820-4706-bef8-e4cf2732a389" (UID: "6ff6c042-8820-4706-bef8-e4cf2732a389"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.675541 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.674987 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-workload-certs" pod "6ff6c042-8820-4706-bef8-e4cf2732a389" (UID: "6ff6c042-8820-4706-bef8-e4cf2732a389"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.675804 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.675779 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ff6c042-8820-4706-bef8-e4cf2732a389-istiod-ca-cert" pod "6ff6c042-8820-4706-bef8-e4cf2732a389" (UID: "6ff6c042-8820-4706-bef8-e4cf2732a389"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:45.676280 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.676232 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff6c042-8820-4706-bef8-e4cf2732a389-istio-token" pod "6ff6c042-8820-4706-bef8-e4cf2732a389" (UID: "6ff6c042-8820-4706-bef8-e4cf2732a389"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:45.676890 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.676841 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-istio-envoy" pod "8210f11c-5046-4e0b-a3ca-d859918e76c7" (UID: "8210f11c-5046-4e0b-a3ca-d859918e76c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.677098 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.677068 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ff6c042-8820-4706-bef8-e4cf2732a389-istio-podinfo" pod "6ff6c042-8820-4706-bef8-e4cf2732a389" (UID: "6ff6c042-8820-4706-bef8-e4cf2732a389"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:11:45.678276 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.678244 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-istio-envoy" pod "6ff6c042-8820-4706-bef8-e4cf2732a389" (UID: "6ff6c042-8820-4706-bef8-e4cf2732a389"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.678817 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.678511 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-credential-socket" pod "8210f11c-5046-4e0b-a3ca-d859918e76c7" (UID: "8210f11c-5046-4e0b-a3ca-d859918e76c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.680499 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.679236 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff6c042-8820-4706-bef8-e4cf2732a389-kube-api-access-hc4x9" pod "6ff6c042-8820-4706-bef8-e4cf2732a389" (UID: "6ff6c042-8820-4706-bef8-e4cf2732a389"). InnerVolumeSpecName "kube-api-access-hc4x9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:45.682398 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.681663 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8210f11c-5046-4e0b-a3ca-d859918e76c7-kube-api-access-x4hxm" pod "8210f11c-5046-4e0b-a3ca-d859918e76c7" (UID: "8210f11c-5046-4e0b-a3ca-d859918e76c7"). InnerVolumeSpecName "kube-api-access-x4hxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:45.682398 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.681769 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8210f11c-5046-4e0b-a3ca-d859918e76c7-istio-token" pod "8210f11c-5046-4e0b-a3ca-d859918e76c7" (UID: "8210f11c-5046-4e0b-a3ca-d859918e76c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:45.683843 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.683816 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8210f11c-5046-4e0b-a3ca-d859918e76c7-istio-podinfo" pod "8210f11c-5046-4e0b-a3ca-d859918e76c7" (UID: "8210f11c-5046-4e0b-a3ca-d859918e76c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:11:45.684175 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.684150 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 18:11:45.835642 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.835594 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-istio-data\") pod \"ef139666-59eb-4847-9f3d-8fc0d20529dc\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " Nov 02 18:11:45.835887 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.835753 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef139666-59eb-4847-9f3d-8fc0d20529dc-istio-token\" (UniqueName: \"kubernetes.io/projected/ef139666-59eb-4847-9f3d-8fc0d20529dc-istio-token\") pod \"ef139666-59eb-4847-9f3d-8fc0d20529dc\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " Nov 02 18:11:45.835887 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.835859 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-istio-envoy\") pod \"ef139666-59eb-4847-9f3d-8fc0d20529dc\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " Nov 02 18:11:45.836043 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.835896 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-workload-socket\") pod \"ef139666-59eb-4847-9f3d-8fc0d20529dc\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " Nov 02 18:11:45.836043 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.835989 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef139666-59eb-4847-9f3d-8fc0d20529dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef139666-59eb-4847-9f3d-8fc0d20529dc-istiod-ca-cert\") pod \"ef139666-59eb-4847-9f3d-8fc0d20529dc\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " Nov 02 18:11:45.836163 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.836075 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-workload-certs\") pod \"ef139666-59eb-4847-9f3d-8fc0d20529dc\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " Nov 02 18:11:45.836163 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.836112 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef139666-59eb-4847-9f3d-8fc0d20529dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef139666-59eb-4847-9f3d-8fc0d20529dc-istio-podinfo\") pod \"ef139666-59eb-4847-9f3d-8fc0d20529dc\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " Nov 02 18:11:45.836296 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.836160 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef139666-59eb-4847-9f3d-8fc0d20529dc-kube-api-access-gc9n4\" (UniqueName: \"kubernetes.io/projected/ef139666-59eb-4847-9f3d-8fc0d20529dc-kube-api-access-gc9n4\") pod \"ef139666-59eb-4847-9f3d-8fc0d20529dc\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " Nov 02 18:11:45.836296 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.836222 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-credential-socket\") pod \"ef139666-59eb-4847-9f3d-8fc0d20529dc\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " Nov 02 18:11:45.870336 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.839700 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-workload-socket" pod "ef139666-59eb-4847-9f3d-8fc0d20529dc" (UID: "ef139666-59eb-4847-9f3d-8fc0d20529dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.870336 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.839940 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-istio-data" pod "ef139666-59eb-4847-9f3d-8fc0d20529dc" (UID: "ef139666-59eb-4847-9f3d-8fc0d20529dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.870336 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.846418 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-workload-certs" pod "ef139666-59eb-4847-9f3d-8fc0d20529dc" (UID: "ef139666-59eb-4847-9f3d-8fc0d20529dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.870336 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.846892 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef139666-59eb-4847-9f3d-8fc0d20529dc-istiod-ca-cert" pod "ef139666-59eb-4847-9f3d-8fc0d20529dc" (UID: "ef139666-59eb-4847-9f3d-8fc0d20529dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:45.870336 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.851103 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-credential-socket" pod "ef139666-59eb-4847-9f3d-8fc0d20529dc" (UID: "ef139666-59eb-4847-9f3d-8fc0d20529dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.870336 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.860759 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4hxm\" (UniqueName: \"kubernetes.io/projected/8210f11c-5046-4e0b-a3ca-d859918e76c7-kube-api-access-x4hxm\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.870336 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.860793 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.870336 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.860809 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff6c042-8820-4706-bef8-e4cf2732a389-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.870336 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.860829 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.870336 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.860843 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.870336 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.860856 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff6c042-8820-4706-bef8-e4cf2732a389-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.870336 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.860871 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc4x9\" (UniqueName: \"kubernetes.io/projected/6ff6c042-8820-4706-bef8-e4cf2732a389-kube-api-access-hc4x9\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.870336 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.860889 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff6c042-8820-4706-bef8-e4cf2732a389-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.870336 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.860904 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8210f11c-5046-4e0b-a3ca-d859918e76c7-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.870336 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.860923 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.870336 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.860937 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff6c042-8820-4706-bef8-e4cf2732a389-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.870336 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.860955 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8210f11c-5046-4e0b-a3ca-d859918e76c7-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.870336 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.860968 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8210f11c-5046-4e0b-a3ca-d859918e76c7-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.936112 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.936025 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn"] Nov 02 18:11:45.936865 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.936798 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef139666-59eb-4847-9f3d-8fc0d20529dc-istio-podinfo" pod "ef139666-59eb-4847-9f3d-8fc0d20529dc" (UID: "ef139666-59eb-4847-9f3d-8fc0d20529dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:11:45.947906 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.941599 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-istio-envoy" pod "ef139666-59eb-4847-9f3d-8fc0d20529dc" (UID: "ef139666-59eb-4847-9f3d-8fc0d20529dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.952689 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.952652 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef139666-59eb-4847-9f3d-8fc0d20529dc-kube-api-access-gc9n4" pod "ef139666-59eb-4847-9f3d-8fc0d20529dc" (UID: "ef139666-59eb-4847-9f3d-8fc0d20529dc"). InnerVolumeSpecName "kube-api-access-gc9n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:45.956441 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.956412 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vbtrn"] Nov 02 18:11:45.961817 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.961773 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef139666-59eb-4847-9f3d-8fc0d20529dc-istio-token" pod "ef139666-59eb-4847-9f3d-8fc0d20529dc" (UID: "ef139666-59eb-4847-9f3d-8fc0d20529dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:45.961999 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.961973 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef139666-59eb-4847-9f3d-8fc0d20529dc-istio-token\" (UniqueName: \"kubernetes.io/projected/ef139666-59eb-4847-9f3d-8fc0d20529dc-istio-token\") pod \"ef139666-59eb-4847-9f3d-8fc0d20529dc\" (UID: \"ef139666-59eb-4847-9f3d-8fc0d20529dc\") " Nov 02 18:11:45.962763 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.962723 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.962854 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.962769 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.962854 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.962784 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef139666-59eb-4847-9f3d-8fc0d20529dc-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.962854 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.962798 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.962854 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.962828 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef139666-59eb-4847-9f3d-8fc0d20529dc-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.962854 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.962841 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc9n4\" (UniqueName: \"kubernetes.io/projected/ef139666-59eb-4847-9f3d-8fc0d20529dc-kube-api-access-gc9n4\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.962854 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.962854 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.963183 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.962867 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef139666-59eb-4847-9f3d-8fc0d20529dc-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.963183 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:11:45.962967 4471 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/ef139666-59eb-4847-9f3d-8fc0d20529dc/volumes/kubernetes.io~projected/istio-token Nov 02 18:11:45.963183 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:45.963000 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef139666-59eb-4847-9f3d-8fc0d20529dc-istio-token" pod "ef139666-59eb-4847-9f3d-8fc0d20529dc" (UID: "ef139666-59eb-4847-9f3d-8fc0d20529dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:46.071483 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.065676 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef139666-59eb-4847-9f3d-8fc0d20529dc-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:46.322756 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.322725 4471 generic.go:356] "Generic (PLEG): container finished" podID="6ff6c042-8820-4706-bef8-e4cf2732a389" containerID="7ffd35bc256878511d42c8959e6a6f834cd9d83ff47ab3a3ff635817057724ae" exitCode=0 Nov 02 18:11:46.322932 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.322803 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" Nov 02 18:11:46.322932 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.322812 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" event={"ID":"6ff6c042-8820-4706-bef8-e4cf2732a389","Type":"ContainerDied","Data":"7ffd35bc256878511d42c8959e6a6f834cd9d83ff47ab3a3ff635817057724ae"} Nov 02 18:11:46.322932 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.322910 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5" event={"ID":"6ff6c042-8820-4706-bef8-e4cf2732a389","Type":"ContainerDied","Data":"df341392f4bfe7d5960f4a4c2c4f06ad3a507999d22df161d60907d7b6a4f175"} Nov 02 18:11:46.323134 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.322945 4471 scope.go:122] "RemoveContainer" containerID="7ffd35bc256878511d42c8959e6a6f834cd9d83ff47ab3a3ff635817057724ae" Nov 02 18:11:46.325292 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.325270 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" Nov 02 18:11:46.325490 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.325277 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv" event={"ID":"8210f11c-5046-4e0b-a3ca-d859918e76c7","Type":"ContainerDied","Data":"476af9d9e74d5f29f05c3c1122701d9e2496acdb7faf27072385352f929181b3"} Nov 02 18:11:46.326523 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.326502 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" Nov 02 18:11:46.326523 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.326508 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq" event={"ID":"1ac349d9-ff43-493a-8cd2-3bfa0836a8da","Type":"ContainerDied","Data":"1a7d11bdf91182fe9088444154d53e8748ca3da85b6f6f10c9726df6ddc3ffec"} Nov 02 18:11:46.330000 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.329961 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" Nov 02 18:11:46.332360 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.329954 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz" event={"ID":"ef139666-59eb-4847-9f3d-8fc0d20529dc","Type":"ContainerDied","Data":"9c729e45cc9aa465b0d5dad972651eb2c93afe184d90bc9293c4526711295ade"} Nov 02 18:11:46.336829 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.336742 4471 scope.go:122] "RemoveContainer" containerID="7ffd35bc256878511d42c8959e6a6f834cd9d83ff47ab3a3ff635817057724ae" Nov 02 18:11:46.337083 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:11:46.337055 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ffd35bc256878511d42c8959e6a6f834cd9d83ff47ab3a3ff635817057724ae\": container with ID starting with 7ffd35bc256878511d42c8959e6a6f834cd9d83ff47ab3a3ff635817057724ae not found: ID does not exist" containerID="7ffd35bc256878511d42c8959e6a6f834cd9d83ff47ab3a3ff635817057724ae" Nov 02 18:11:46.337171 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.337095 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ffd35bc256878511d42c8959e6a6f834cd9d83ff47ab3a3ff635817057724ae"} err="failed to get container status \"7ffd35bc256878511d42c8959e6a6f834cd9d83ff47ab3a3ff635817057724ae\": rpc error: code = NotFound desc = could not find container \"7ffd35bc256878511d42c8959e6a6f834cd9d83ff47ab3a3ff635817057724ae\": container with ID starting with 7ffd35bc256878511d42c8959e6a6f834cd9d83ff47ab3a3ff635817057724ae not found: ID does not exist" Nov 02 18:11:46.337171 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.337119 4471 scope.go:122] "RemoveContainer" containerID="6fb440c60b1ae9dabe51a4b1c3170b9d9f39b060fa910f0af37932cb43dacdd8" Nov 02 18:11:46.349642 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.349626 4471 scope.go:122] "RemoveContainer" containerID="63daf3d4ceb7310bf60553eb303ae9de8030019b0e38713356b0a95fe2bfc420" Nov 02 18:11:46.428298 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.428224 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv"] Nov 02 18:11:46.432944 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.432920 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-nsvrv"] Nov 02 18:11:46.441964 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.441941 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24f05804-b83e-44f0-b01d-b1f34904fba2" path="/var/lib/kubelet/pods/24f05804-b83e-44f0-b01d-b1f34904fba2/volumes" Nov 02 18:11:46.442601 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.442582 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38fea887-fcc0-42fa-a1c9-320e44f70562" path="/var/lib/kubelet/pods/38fea887-fcc0-42fa-a1c9-320e44f70562/volumes" Nov 02 18:11:46.443102 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.443088 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8210f11c-5046-4e0b-a3ca-d859918e76c7" path="/var/lib/kubelet/pods/8210f11c-5046-4e0b-a3ca-d859918e76c7/volumes" Nov 02 18:11:46.444000 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.443985 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e881e4aa-a5ad-46d2-952d-97c5118e98b3" path="/var/lib/kubelet/pods/e881e4aa-a5ad-46d2-952d-97c5118e98b3/volumes" Nov 02 18:11:46.444472 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.444458 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8eeabf7-f670-46c2-bb59-e07e1a1f1337" path="/var/lib/kubelet/pods/e8eeabf7-f670-46c2-bb59-e07e1a1f1337/volumes" Nov 02 18:11:46.468400 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.468369 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz"] Nov 02 18:11:46.477708 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.477683 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-l9pgz"] Nov 02 18:11:46.511879 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.511849 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5"] Nov 02 18:11:46.517663 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.517631 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-s22d5"] Nov 02 18:11:46.552405 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.552375 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq"] Nov 02 18:11:46.559118 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.559090 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-gvtpq"] Nov 02 18:11:46.731415 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.731334 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv"] Nov 02 18:11:46.731768 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.731595 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" podUID="2c047abd-2c48-4b50-ba99-579d0f2cc3fd" containerName="istio-proxy" containerID="cri-o://d5458af601b3e44d264665e61a48bcf0f5f2493bbb40b4dbac1f9bd29c16d58e" gracePeriod=30 Nov 02 18:11:46.877906 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.877861 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk"] Nov 02 18:11:46.881355 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.877958 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc"] Nov 02 18:11:46.881565 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.881537 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" podUID="9ebaee6a-d3f4-4f0d-8485-6dd565c1e078" containerName="istio-proxy" containerID="cri-o://3bb3b7e01e995024c94215d5661cd6350bed5f90ed1c81bb2169c1a1544fdc66" gracePeriod=30 Nov 02 18:11:46.881700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.880493 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" podUID="d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5" containerName="istio-proxy" containerID="cri-o://032035c86e70ca74d40491c92948cb2032b9912590738e4442690c6427481490" gracePeriod=30 Nov 02 18:11:46.885919 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.885878 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46"] Nov 02 18:11:46.886136 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.886108 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" podUID="cd908c8a-4026-4da6-95c5-34112ecc019a" containerName="istio-proxy" containerID="cri-o://dd1da4ad3c90b90be0ec428ffbd80ac1faffbdaca8fedfc44005142de19bc009" gracePeriod=30 Nov 02 18:11:46.894928 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.894897 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw"] Nov 02 18:11:46.895103 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:46.895082 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" podUID="1aa74f62-fe2f-4238-9f58-4a0a20f7892a" containerName="istio-proxy" containerID="cri-o://23a9aaf637c69775424d91c36553ff299b37381bff46db3b401612c0853ec643" gracePeriod=30 Nov 02 18:11:47.010297 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:47.009553 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-7l8nw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:11:47.010297 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:47.010260 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" podUID="1aa74f62-fe2f-4238-9f58-4a0a20f7892a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:11:48.442912 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:48.442874 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ac349d9-ff43-493a-8cd2-3bfa0836a8da" path="/var/lib/kubelet/pods/1ac349d9-ff43-493a-8cd2-3bfa0836a8da/volumes" Nov 02 18:11:48.443464 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:48.443449 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ff6c042-8820-4706-bef8-e4cf2732a389" path="/var/lib/kubelet/pods/6ff6c042-8820-4706-bef8-e4cf2732a389/volumes" Nov 02 18:11:48.443908 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:48.443895 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef139666-59eb-4847-9f3d-8fc0d20529dc" path="/var/lib/kubelet/pods/ef139666-59eb-4847-9f3d-8fc0d20529dc/volumes" Nov 02 18:11:48.660796 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:48.660758 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-k8n46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:11:48.660947 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:48.660836 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" podUID="cd908c8a-4026-4da6-95c5-34112ecc019a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:11:49.368094 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:49.368039 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gg9fk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:11:49.368273 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:49.368110 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" podUID="d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:11:49.994238 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:49.994195 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a209f-789b5bc59f-lgg4z" podUID="dd2eb7cb-d5ff-4351-84ac-f19fabf7e616" containerName="payload-pre-processing" containerID="cri-o://47345402a5b9cb8bb2b10d5ba64e407fc614e603dc190b7b317dbefb51ca51a6" gracePeriod=25 Nov 02 18:11:51.313204 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:51.313173 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b91583-66bb7b5959-5596q"] Nov 02 18:11:52.148152 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.148057 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-9b8d5bbf-xnjnp" podUID="83085553-108f-4d34-b8de-9d0c6e0b13e5" containerName="console" containerID="cri-o://773191a50fdcf4ae1309a440e1737b7705be93ba616179653ccf5cb035fd7e22" gracePeriod=15 Nov 02 18:11:52.195685 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:11:52.195641 4471 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ebaee6a_d3f4_4f0d_8485_6dd565c1e078.slice/crio-3bb3b7e01e995024c94215d5661cd6350bed5f90ed1c81bb2169c1a1544fdc66.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd14d1a7f_1dc5_4a6d_893b_03b53bc1cce5.slice/crio-032035c86e70ca74d40491c92948cb2032b9912590738e4442690c6427481490.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd908c8a_4026_4da6_95c5_34112ecc019a.slice/crio-conmon-dd1da4ad3c90b90be0ec428ffbd80ac1faffbdaca8fedfc44005142de19bc009.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd908c8a_4026_4da6_95c5_34112ecc019a.slice/crio-dd1da4ad3c90b90be0ec428ffbd80ac1faffbdaca8fedfc44005142de19bc009.scope\": RecentStats: unable to find data in memory cache]" Nov 02 18:11:52.331882 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.331855 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 18:11:52.366143 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.364478 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istiod-ca-cert\") pod \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " Nov 02 18:11:52.366143 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.364537 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-kube-api-access-sgvrz\" (UniqueName: \"kubernetes.io/projected/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-kube-api-access-sgvrz\") pod \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " Nov 02 18:11:52.366143 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.364634 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-workload-certs\") pod \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " Nov 02 18:11:52.366143 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.364719 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-credential-socket\") pod \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " Nov 02 18:11:52.366143 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.364768 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istio-token\" (UniqueName: \"kubernetes.io/projected/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istio-token\") pod \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " Nov 02 18:11:52.366143 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.364822 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istio-podinfo\") pod \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " Nov 02 18:11:52.366143 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.364848 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istio-data\") pod \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " Nov 02 18:11:52.366143 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.364914 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istio-envoy\") pod \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " Nov 02 18:11:52.366143 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.365049 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-workload-socket\") pod \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\" (UID: \"2c047abd-2c48-4b50-ba99-579d0f2cc3fd\") " Nov 02 18:11:52.367216 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.366913 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-credential-socket" pod "2c047abd-2c48-4b50-ba99-579d0f2cc3fd" (UID: "2c047abd-2c48-4b50-ba99-579d0f2cc3fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.367216 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.366944 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-workload-socket" pod "2c047abd-2c48-4b50-ba99-579d0f2cc3fd" (UID: "2c047abd-2c48-4b50-ba99-579d0f2cc3fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.367908 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.367624 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istiod-ca-cert" pod "2c047abd-2c48-4b50-ba99-579d0f2cc3fd" (UID: "2c047abd-2c48-4b50-ba99-579d0f2cc3fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:52.368710 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.368688 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istio-data" pod "2c047abd-2c48-4b50-ba99-579d0f2cc3fd" (UID: "2c047abd-2c48-4b50-ba99-579d0f2cc3fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.369065 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.369025 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-workload-certs" pod "2c047abd-2c48-4b50-ba99-579d0f2cc3fd" (UID: "2c047abd-2c48-4b50-ba99-579d0f2cc3fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.369927 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.369898 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istio-token" pod "2c047abd-2c48-4b50-ba99-579d0f2cc3fd" (UID: "2c047abd-2c48-4b50-ba99-579d0f2cc3fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:52.370029 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.370007 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-kube-api-access-sgvrz" pod "2c047abd-2c48-4b50-ba99-579d0f2cc3fd" (UID: "2c047abd-2c48-4b50-ba99-579d0f2cc3fd"). InnerVolumeSpecName "kube-api-access-sgvrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:52.371008 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.370980 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istio-podinfo" pod "2c047abd-2c48-4b50-ba99-579d0f2cc3fd" (UID: "2c047abd-2c48-4b50-ba99-579d0f2cc3fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:11:52.371760 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.371731 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istio-envoy" pod "2c047abd-2c48-4b50-ba99-579d0f2cc3fd" (UID: "2c047abd-2c48-4b50-ba99-579d0f2cc3fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.424566 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.424299 4471 generic.go:356] "Generic (PLEG): container finished" podID="9ebaee6a-d3f4-4f0d-8485-6dd565c1e078" containerID="3bb3b7e01e995024c94215d5661cd6350bed5f90ed1c81bb2169c1a1544fdc66" exitCode=0 Nov 02 18:11:52.424566 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.424490 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" event={"ID":"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078","Type":"ContainerDied","Data":"3bb3b7e01e995024c94215d5661cd6350bed5f90ed1c81bb2169c1a1544fdc66"} Nov 02 18:11:52.431878 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.431648 4471 generic.go:356] "Generic (PLEG): container finished" podID="2c047abd-2c48-4b50-ba99-579d0f2cc3fd" containerID="d5458af601b3e44d264665e61a48bcf0f5f2493bbb40b4dbac1f9bd29c16d58e" exitCode=0 Nov 02 18:11:52.431878 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.431764 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" event={"ID":"2c047abd-2c48-4b50-ba99-579d0f2cc3fd","Type":"ContainerDied","Data":"d5458af601b3e44d264665e61a48bcf0f5f2493bbb40b4dbac1f9bd29c16d58e"} Nov 02 18:11:52.431878 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.431820 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" event={"ID":"2c047abd-2c48-4b50-ba99-579d0f2cc3fd","Type":"ContainerDied","Data":"9d4897367513b6428dc82f2b1589f1e36ceabe23269ab90f85c2edd20e5cf023"} Nov 02 18:11:52.435290 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.432935 4471 scope.go:122] "RemoveContainer" containerID="d5458af601b3e44d264665e61a48bcf0f5f2493bbb40b4dbac1f9bd29c16d58e" Nov 02 18:11:52.435290 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.433120 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv" Nov 02 18:11:52.441724 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.441620 4471 generic.go:356] "Generic (PLEG): container finished" podID="cd908c8a-4026-4da6-95c5-34112ecc019a" containerID="dd1da4ad3c90b90be0ec428ffbd80ac1faffbdaca8fedfc44005142de19bc009" exitCode=0 Nov 02 18:11:52.444701 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.443405 4471 generic.go:356] "Generic (PLEG): container finished" podID="83085553-108f-4d34-b8de-9d0c6e0b13e5" containerID="773191a50fdcf4ae1309a440e1737b7705be93ba616179653ccf5cb035fd7e22" exitCode=0 Nov 02 18:11:52.448380 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.446781 4471 generic.go:356] "Generic (PLEG): container finished" podID="1aa74f62-fe2f-4238-9f58-4a0a20f7892a" containerID="23a9aaf637c69775424d91c36553ff299b37381bff46db3b401612c0853ec643" exitCode=0 Nov 02 18:11:52.448701 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.448677 4471 generic.go:356] "Generic (PLEG): container finished" podID="d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5" containerID="032035c86e70ca74d40491c92948cb2032b9912590738e4442690c6427481490" exitCode=0 Nov 02 18:11:52.478333 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.476044 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.478333 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.476076 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.478333 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.476093 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.478333 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.476119 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgvrz\" (UniqueName: \"kubernetes.io/projected/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-kube-api-access-sgvrz\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.478333 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.476138 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.478333 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.476154 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.478333 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.476169 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.478333 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.476187 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.478333 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.476199 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c047abd-2c48-4b50-ba99-579d0f2cc3fd-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.488117 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.488089 4471 scope.go:122] "RemoveContainer" containerID="15d32558f6a2658c05e1570f0d584c3a07008d0a460897a472f0f80216cfe32b" Nov 02 18:11:52.492132 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.492097 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" event={"ID":"cd908c8a-4026-4da6-95c5-34112ecc019a","Type":"ContainerDied","Data":"dd1da4ad3c90b90be0ec428ffbd80ac1faffbdaca8fedfc44005142de19bc009"} Nov 02 18:11:52.492379 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.492345 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9b8d5bbf-xnjnp" event={"ID":"83085553-108f-4d34-b8de-9d0c6e0b13e5","Type":"ContainerDied","Data":"773191a50fdcf4ae1309a440e1737b7705be93ba616179653ccf5cb035fd7e22"} Nov 02 18:11:52.492520 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.492505 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" event={"ID":"1aa74f62-fe2f-4238-9f58-4a0a20f7892a","Type":"ContainerDied","Data":"23a9aaf637c69775424d91c36553ff299b37381bff46db3b401612c0853ec643"} Nov 02 18:11:52.492650 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.492637 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" event={"ID":"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5","Type":"ContainerDied","Data":"032035c86e70ca74d40491c92948cb2032b9912590738e4442690c6427481490"} Nov 02 18:11:52.529026 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.528975 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv"] Nov 02 18:11:52.533633 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.533616 4471 scope.go:122] "RemoveContainer" containerID="d5458af601b3e44d264665e61a48bcf0f5f2493bbb40b4dbac1f9bd29c16d58e" Nov 02 18:11:52.534078 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:11:52.534059 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5458af601b3e44d264665e61a48bcf0f5f2493bbb40b4dbac1f9bd29c16d58e\": container with ID starting with d5458af601b3e44d264665e61a48bcf0f5f2493bbb40b4dbac1f9bd29c16d58e not found: ID does not exist" containerID="d5458af601b3e44d264665e61a48bcf0f5f2493bbb40b4dbac1f9bd29c16d58e" Nov 02 18:11:52.534201 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.534179 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5458af601b3e44d264665e61a48bcf0f5f2493bbb40b4dbac1f9bd29c16d58e"} err="failed to get container status \"d5458af601b3e44d264665e61a48bcf0f5f2493bbb40b4dbac1f9bd29c16d58e\": rpc error: code = NotFound desc = could not find container \"d5458af601b3e44d264665e61a48bcf0f5f2493bbb40b4dbac1f9bd29c16d58e\": container with ID starting with d5458af601b3e44d264665e61a48bcf0f5f2493bbb40b4dbac1f9bd29c16d58e not found: ID does not exist" Nov 02 18:11:52.534690 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.534352 4471 scope.go:122] "RemoveContainer" containerID="15d32558f6a2658c05e1570f0d584c3a07008d0a460897a472f0f80216cfe32b" Nov 02 18:11:52.535514 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:11:52.535471 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15d32558f6a2658c05e1570f0d584c3a07008d0a460897a472f0f80216cfe32b\": container with ID starting with 15d32558f6a2658c05e1570f0d584c3a07008d0a460897a472f0f80216cfe32b not found: ID does not exist" containerID="15d32558f6a2658c05e1570f0d584c3a07008d0a460897a472f0f80216cfe32b" Nov 02 18:11:52.535514 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.535500 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15d32558f6a2658c05e1570f0d584c3a07008d0a460897a472f0f80216cfe32b"} err="failed to get container status \"15d32558f6a2658c05e1570f0d584c3a07008d0a460897a472f0f80216cfe32b\": rpc error: code = NotFound desc = could not find container \"15d32558f6a2658c05e1570f0d584c3a07008d0a460897a472f0f80216cfe32b\": container with ID starting with 15d32558f6a2658c05e1570f0d584c3a07008d0a460897a472f0f80216cfe32b not found: ID does not exist" Nov 02 18:11:52.536298 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.536271 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-9kwhv"] Nov 02 18:11:52.560696 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.560670 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p" Nov 02 18:11:52.580669 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.580550 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 18:11:52.580797 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.580683 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p" Nov 02 18:11:52.585035 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.585008 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 18:11:52.588494 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.588467 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 18:11:52.592060 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.592036 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 18:11:52.679713 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.679678 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-workload-socket\") pod \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " Nov 02 18:11:52.679855 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.679749 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istio-podinfo\") pod \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " Nov 02 18:11:52.679855 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.679779 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd908c8a-4026-4da6-95c5-34112ecc019a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd908c8a-4026-4da6-95c5-34112ecc019a-istio-podinfo\") pod \"cd908c8a-4026-4da6-95c5-34112ecc019a\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " Nov 02 18:11:52.679855 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.679844 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-credential-socket\") pod \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " Nov 02 18:11:52.680048 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.679882 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-kube-api-access-szssp\" (UniqueName: \"kubernetes.io/projected/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-kube-api-access-szssp\") pod \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " Nov 02 18:11:52.680048 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.679961 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istio-podinfo\") pod \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " Nov 02 18:11:52.680048 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.679993 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istio-data\") pod \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " Nov 02 18:11:52.680048 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.680025 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-workload-certs\") pod \"cd908c8a-4026-4da6-95c5-34112ecc019a\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " Nov 02 18:11:52.680277 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.680058 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-credential-socket\") pod \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " Nov 02 18:11:52.680277 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.680089 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-workload-certs\") pod \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " Nov 02 18:11:52.680277 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.680121 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istio-envoy\") pod \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " Nov 02 18:11:52.680277 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.680155 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istio-podinfo\") pod \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " Nov 02 18:11:52.680277 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.680208 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istio-data\") pod \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " Nov 02 18:11:52.680277 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.680267 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-workload-certs\") pod \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " Nov 02 18:11:52.680621 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.680368 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istio-data\") pod \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " Nov 02 18:11:52.680621 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.680410 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istio-token\" (UniqueName: \"kubernetes.io/projected/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istio-token\") pod \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " Nov 02 18:11:52.680621 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.680437 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istio-envoy\") pod \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " Nov 02 18:11:52.680621 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.680467 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd908c8a-4026-4da6-95c5-34112ecc019a-istio-token\" (UniqueName: \"kubernetes.io/projected/cd908c8a-4026-4da6-95c5-34112ecc019a-istio-token\") pod \"cd908c8a-4026-4da6-95c5-34112ecc019a\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " Nov 02 18:11:52.680621 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.680504 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd908c8a-4026-4da6-95c5-34112ecc019a-kube-api-access-m4c6k\" (UniqueName: \"kubernetes.io/projected/cd908c8a-4026-4da6-95c5-34112ecc019a-kube-api-access-m4c6k\") pod \"cd908c8a-4026-4da6-95c5-34112ecc019a\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " Nov 02 18:11:52.680621 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.680536 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-workload-socket\") pod \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " Nov 02 18:11:52.680621 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.680589 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-workload-certs\") pod \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " Nov 02 18:11:52.680621 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.680619 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-workload-socket\") pod \"cd908c8a-4026-4da6-95c5-34112ecc019a\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " Nov 02 18:11:52.681224 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.680683 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-credential-socket\") pod \"cd908c8a-4026-4da6-95c5-34112ecc019a\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " Nov 02 18:11:52.681224 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.680721 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istiod-ca-cert\") pod \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " Nov 02 18:11:52.681224 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.680754 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-istio-envoy\") pod \"cd908c8a-4026-4da6-95c5-34112ecc019a\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " Nov 02 18:11:52.681224 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.680798 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istiod-ca-cert\") pod \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " Nov 02 18:11:52.681224 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.680828 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istiod-ca-cert\") pod \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " Nov 02 18:11:52.681224 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.680876 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-kube-api-access-p6cv2\" (UniqueName: \"kubernetes.io/projected/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-kube-api-access-p6cv2\") pod \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " Nov 02 18:11:52.681224 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.680910 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istio-token\" (UniqueName: \"kubernetes.io/projected/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istio-token\") pod \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " Nov 02 18:11:52.681224 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.680939 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-workload-socket\") pod \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " Nov 02 18:11:52.681224 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.681012 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istio-token\" (UniqueName: \"kubernetes.io/projected/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istio-token\") pod \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " Nov 02 18:11:52.681224 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.681053 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-credential-socket\") pod \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\" (UID: \"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078\") " Nov 02 18:11:52.681224 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.681110 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-kube-api-access-vv6xv\" (UniqueName: \"kubernetes.io/projected/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-kube-api-access-vv6xv\") pod \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\" (UID: \"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5\") " Nov 02 18:11:52.681224 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.681189 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-istio-data\") pod \"cd908c8a-4026-4da6-95c5-34112ecc019a\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " Nov 02 18:11:52.682400 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.681233 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istio-envoy\") pod \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\" (UID: \"1aa74f62-fe2f-4238-9f58-4a0a20f7892a\") " Nov 02 18:11:52.682400 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.681287 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd908c8a-4026-4da6-95c5-34112ecc019a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd908c8a-4026-4da6-95c5-34112ecc019a-istiod-ca-cert\") pod \"cd908c8a-4026-4da6-95c5-34112ecc019a\" (UID: \"cd908c8a-4026-4da6-95c5-34112ecc019a\") " Nov 02 18:11:52.682400 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.681812 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-workload-certs" pod "1aa74f62-fe2f-4238-9f58-4a0a20f7892a" (UID: "1aa74f62-fe2f-4238-9f58-4a0a20f7892a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.682400 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.682391 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istiod-ca-cert" pod "d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5" (UID: "d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:52.683321 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.682815 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.683321 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.682839 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.683321 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.683172 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-workload-socket" pod "cd908c8a-4026-4da6-95c5-34112ecc019a" (UID: "cd908c8a-4026-4da6-95c5-34112ecc019a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.683527 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.683357 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-credential-socket" pod "cd908c8a-4026-4da6-95c5-34112ecc019a" (UID: "cd908c8a-4026-4da6-95c5-34112ecc019a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.683671 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.683650 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istiod-ca-cert" pod "1aa74f62-fe2f-4238-9f58-4a0a20f7892a" (UID: "1aa74f62-fe2f-4238-9f58-4a0a20f7892a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:52.685886 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.684505 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istio-token" pod "9ebaee6a-d3f4-4f0d-8485-6dd565c1e078" (UID: "9ebaee6a-d3f4-4f0d-8485-6dd565c1e078"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:52.685886 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.684614 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-credential-socket" pod "9ebaee6a-d3f4-4f0d-8485-6dd565c1e078" (UID: "9ebaee6a-d3f4-4f0d-8485-6dd565c1e078"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.685886 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.684918 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istiod-ca-cert" pod "9ebaee6a-d3f4-4f0d-8485-6dd565c1e078" (UID: "9ebaee6a-d3f4-4f0d-8485-6dd565c1e078"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:52.685886 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.685181 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istio-envoy" pod "9ebaee6a-d3f4-4f0d-8485-6dd565c1e078" (UID: "9ebaee6a-d3f4-4f0d-8485-6dd565c1e078"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.685886 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.685439 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-workload-socket" pod "9ebaee6a-d3f4-4f0d-8485-6dd565c1e078" (UID: "9ebaee6a-d3f4-4f0d-8485-6dd565c1e078"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.686802 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.686777 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-workload-certs" pod "cd908c8a-4026-4da6-95c5-34112ecc019a" (UID: "cd908c8a-4026-4da6-95c5-34112ecc019a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.687013 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.686992 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-workload-socket" pod "1aa74f62-fe2f-4238-9f58-4a0a20f7892a" (UID: "1aa74f62-fe2f-4238-9f58-4a0a20f7892a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.687573 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.687271 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-kube-api-access-p6cv2" pod "1aa74f62-fe2f-4238-9f58-4a0a20f7892a" (UID: "1aa74f62-fe2f-4238-9f58-4a0a20f7892a"). InnerVolumeSpecName "kube-api-access-p6cv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:52.688072 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.688048 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd908c8a-4026-4da6-95c5-34112ecc019a-istiod-ca-cert" pod "cd908c8a-4026-4da6-95c5-34112ecc019a" (UID: "cd908c8a-4026-4da6-95c5-34112ecc019a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:52.690450 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.690426 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istio-podinfo" pod "1aa74f62-fe2f-4238-9f58-4a0a20f7892a" (UID: "1aa74f62-fe2f-4238-9f58-4a0a20f7892a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:11:52.692265 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.692231 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-credential-socket" pod "d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5" (UID: "d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.692503 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.692482 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-workload-certs" pod "9ebaee6a-d3f4-4f0d-8485-6dd565c1e078" (UID: "9ebaee6a-d3f4-4f0d-8485-6dd565c1e078"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.692780 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.692755 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd908c8a-4026-4da6-95c5-34112ecc019a-istio-podinfo" pod "cd908c8a-4026-4da6-95c5-34112ecc019a" (UID: "cd908c8a-4026-4da6-95c5-34112ecc019a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:11:52.692984 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.692956 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-kube-api-access-vv6xv" pod "d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5" (UID: "d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5"). InnerVolumeSpecName "kube-api-access-vv6xv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:52.692984 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.692976 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-credential-socket" pod "1aa74f62-fe2f-4238-9f58-4a0a20f7892a" (UID: "1aa74f62-fe2f-4238-9f58-4a0a20f7892a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.693188 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.693168 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istio-data" pod "9ebaee6a-d3f4-4f0d-8485-6dd565c1e078" (UID: "9ebaee6a-d3f4-4f0d-8485-6dd565c1e078"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.693269 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.693253 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istio-token" pod "d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5" (UID: "d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:52.693490 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.693468 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-workload-socket" pod "d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5" (UID: "d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.693685 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.693661 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-istio-data" pod "cd908c8a-4026-4da6-95c5-34112ecc019a" (UID: "cd908c8a-4026-4da6-95c5-34112ecc019a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.693888 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.693869 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istio-data" pod "d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5" (UID: "d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.694054 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.694032 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-workload-certs" pod "d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5" (UID: "d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.694236 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.694216 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istio-data" pod "1aa74f62-fe2f-4238-9f58-4a0a20f7892a" (UID: "1aa74f62-fe2f-4238-9f58-4a0a20f7892a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.694392 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.694271 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9b8d5bbf-xnjnp" Nov 02 18:11:52.694856 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.694823 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istio-token" pod "1aa74f62-fe2f-4238-9f58-4a0a20f7892a" (UID: "1aa74f62-fe2f-4238-9f58-4a0a20f7892a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:52.698419 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.695847 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-istio-envoy" pod "cd908c8a-4026-4da6-95c5-34112ecc019a" (UID: "cd908c8a-4026-4da6-95c5-34112ecc019a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.698419 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.695850 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd908c8a-4026-4da6-95c5-34112ecc019a-istio-token" pod "cd908c8a-4026-4da6-95c5-34112ecc019a" (UID: "cd908c8a-4026-4da6-95c5-34112ecc019a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:52.698419 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.695922 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istio-envoy" pod "1aa74f62-fe2f-4238-9f58-4a0a20f7892a" (UID: "1aa74f62-fe2f-4238-9f58-4a0a20f7892a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.698419 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.695981 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd908c8a-4026-4da6-95c5-34112ecc019a-kube-api-access-m4c6k" pod "cd908c8a-4026-4da6-95c5-34112ecc019a" (UID: "cd908c8a-4026-4da6-95c5-34112ecc019a"). InnerVolumeSpecName "kube-api-access-m4c6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:52.698419 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.696168 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istio-podinfo" pod "9ebaee6a-d3f4-4f0d-8485-6dd565c1e078" (UID: "9ebaee6a-d3f4-4f0d-8485-6dd565c1e078"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:11:52.699333 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.699287 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-kube-api-access-szssp" pod "9ebaee6a-d3f4-4f0d-8485-6dd565c1e078" (UID: "9ebaee6a-d3f4-4f0d-8485-6dd565c1e078"). InnerVolumeSpecName "kube-api-access-szssp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:52.699742 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.699715 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istio-podinfo" pod "d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5" (UID: "d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:11:52.699742 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.699728 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istio-envoy" pod "d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5" (UID: "d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.783699 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.783640 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83085553-108f-4d34-b8de-9d0c6e0b13e5-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83085553-108f-4d34-b8de-9d0c6e0b13e5-console-oauth-config\") pod \"83085553-108f-4d34-b8de-9d0c6e0b13e5\" (UID: \"83085553-108f-4d34-b8de-9d0c6e0b13e5\") " Nov 02 18:11:52.783848 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.783803 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83085553-108f-4d34-b8de-9d0c6e0b13e5-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83085553-108f-4d34-b8de-9d0c6e0b13e5-trusted-ca-bundle\") pod \"83085553-108f-4d34-b8de-9d0c6e0b13e5\" (UID: \"83085553-108f-4d34-b8de-9d0c6e0b13e5\") " Nov 02 18:11:52.783913 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.783845 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83085553-108f-4d34-b8de-9d0c6e0b13e5-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83085553-108f-4d34-b8de-9d0c6e0b13e5-console-serving-cert\") pod \"83085553-108f-4d34-b8de-9d0c6e0b13e5\" (UID: \"83085553-108f-4d34-b8de-9d0c6e0b13e5\") " Nov 02 18:11:52.783913 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.783902 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83085553-108f-4d34-b8de-9d0c6e0b13e5-tmp\" (UniqueName: \"kubernetes.io/empty-dir/83085553-108f-4d34-b8de-9d0c6e0b13e5-tmp\") pod \"83085553-108f-4d34-b8de-9d0c6e0b13e5\" (UID: \"83085553-108f-4d34-b8de-9d0c6e0b13e5\") " Nov 02 18:11:52.784042 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.783970 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83085553-108f-4d34-b8de-9d0c6e0b13e5-service-ca\" (UniqueName: \"kubernetes.io/configmap/83085553-108f-4d34-b8de-9d0c6e0b13e5-service-ca\") pod \"83085553-108f-4d34-b8de-9d0c6e0b13e5\" (UID: \"83085553-108f-4d34-b8de-9d0c6e0b13e5\") " Nov 02 18:11:52.784104 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.784073 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83085553-108f-4d34-b8de-9d0c6e0b13e5-kube-api-access-7b9p2\" (UniqueName: \"kubernetes.io/projected/83085553-108f-4d34-b8de-9d0c6e0b13e5-kube-api-access-7b9p2\") pod \"83085553-108f-4d34-b8de-9d0c6e0b13e5\" (UID: \"83085553-108f-4d34-b8de-9d0c6e0b13e5\") " Nov 02 18:11:52.784210 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.784182 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83085553-108f-4d34-b8de-9d0c6e0b13e5-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83085553-108f-4d34-b8de-9d0c6e0b13e5-oauth-serving-cert\") pod \"83085553-108f-4d34-b8de-9d0c6e0b13e5\" (UID: \"83085553-108f-4d34-b8de-9d0c6e0b13e5\") " Nov 02 18:11:52.784353 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.784279 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83085553-108f-4d34-b8de-9d0c6e0b13e5-tmp" pod "83085553-108f-4d34-b8de-9d0c6e0b13e5" (UID: "83085553-108f-4d34-b8de-9d0c6e0b13e5"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.784353 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.784333 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83085553-108f-4d34-b8de-9d0c6e0b13e5-trusted-ca-bundle" pod "83085553-108f-4d34-b8de-9d0c6e0b13e5" (UID: "83085553-108f-4d34-b8de-9d0c6e0b13e5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:52.784494 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.784390 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83085553-108f-4d34-b8de-9d0c6e0b13e5-console-config\" (UniqueName: \"kubernetes.io/configmap/83085553-108f-4d34-b8de-9d0c6e0b13e5-console-config\") pod \"83085553-108f-4d34-b8de-9d0c6e0b13e5\" (UID: \"83085553-108f-4d34-b8de-9d0c6e0b13e5\") " Nov 02 18:11:52.784735 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.784706 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83085553-108f-4d34-b8de-9d0c6e0b13e5-service-ca" pod "83085553-108f-4d34-b8de-9d0c6e0b13e5" (UID: "83085553-108f-4d34-b8de-9d0c6e0b13e5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:52.785055 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785031 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785140 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785059 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785140 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785073 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785140 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785084 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785140 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785096 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785140 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785108 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785140 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785121 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785140 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785132 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785554 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785143 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785554 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785155 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785554 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785165 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785554 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785176 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785554 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785189 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd908c8a-4026-4da6-95c5-34112ecc019a-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785554 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785201 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4c6k\" (UniqueName: \"kubernetes.io/projected/cd908c8a-4026-4da6-95c5-34112ecc019a-kube-api-access-m4c6k\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785554 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785213 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785554 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785225 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785554 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785237 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785554 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785248 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785554 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785260 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785554 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785271 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785554 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785282 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6cv2\" (UniqueName: \"kubernetes.io/projected/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-kube-api-access-p6cv2\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785554 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785295 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785554 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785328 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785554 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785341 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785554 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785353 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785554 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785365 4471 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83085553-108f-4d34-b8de-9d0c6e0b13e5-trusted-ca-bundle\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785554 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785378 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv6xv\" (UniqueName: \"kubernetes.io/projected/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5-kube-api-access-vv6xv\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785554 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785389 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd908c8a-4026-4da6-95c5-34112ecc019a-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785554 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785401 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785554 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785413 4471 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83085553-108f-4d34-b8de-9d0c6e0b13e5-tmp\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785554 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785423 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd908c8a-4026-4da6-95c5-34112ecc019a-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785554 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785435 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785554 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785052 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83085553-108f-4d34-b8de-9d0c6e0b13e5-oauth-serving-cert" pod "83085553-108f-4d34-b8de-9d0c6e0b13e5" (UID: "83085553-108f-4d34-b8de-9d0c6e0b13e5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:52.785554 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785447 4471 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/83085553-108f-4d34-b8de-9d0c6e0b13e5-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785554 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785494 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785554 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785512 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd908c8a-4026-4da6-95c5-34112ecc019a-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785554 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785525 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa74f62-fe2f-4238-9f58-4a0a20f7892a-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.785554 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785541 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szssp\" (UniqueName: \"kubernetes.io/projected/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078-kube-api-access-szssp\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.787157 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.785711 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83085553-108f-4d34-b8de-9d0c6e0b13e5-console-config" pod "83085553-108f-4d34-b8de-9d0c6e0b13e5" (UID: "83085553-108f-4d34-b8de-9d0c6e0b13e5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:52.787157 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.786812 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83085553-108f-4d34-b8de-9d0c6e0b13e5-console-oauth-config" pod "83085553-108f-4d34-b8de-9d0c6e0b13e5" (UID: "83085553-108f-4d34-b8de-9d0c6e0b13e5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:11:52.787448 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.787420 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83085553-108f-4d34-b8de-9d0c6e0b13e5-kube-api-access-7b9p2" pod "83085553-108f-4d34-b8de-9d0c6e0b13e5" (UID: "83085553-108f-4d34-b8de-9d0c6e0b13e5"). InnerVolumeSpecName "kube-api-access-7b9p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:52.787637 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.787616 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83085553-108f-4d34-b8de-9d0c6e0b13e5-console-serving-cert" pod "83085553-108f-4d34-b8de-9d0c6e0b13e5" (UID: "83085553-108f-4d34-b8de-9d0c6e0b13e5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:11:52.886606 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.886520 4471 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83085553-108f-4d34-b8de-9d0c6e0b13e5-console-oauth-config\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.886606 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.886552 4471 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83085553-108f-4d34-b8de-9d0c6e0b13e5-console-serving-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.886606 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.886566 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b9p2\" (UniqueName: \"kubernetes.io/projected/83085553-108f-4d34-b8de-9d0c6e0b13e5-kube-api-access-7b9p2\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.886606 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.886579 4471 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83085553-108f-4d34-b8de-9d0c6e0b13e5-oauth-serving-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.886606 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.886592 4471 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/83085553-108f-4d34-b8de-9d0c6e0b13e5-console-config\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.975078 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.975005 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr"] Nov 02 18:11:52.975386 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:52.975353 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" podUID="74ec749a-738f-4509-a7e5-be6eb97f4777" containerName="istio-proxy" containerID="cri-o://3a8171f7433846b9317402622b74a5b56ead44961ed13c1ef3f5bef281544230" gracePeriod=30 Nov 02 18:11:53.458990 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:53.458957 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" Nov 02 18:11:53.459532 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:53.458962 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46" event={"ID":"cd908c8a-4026-4da6-95c5-34112ecc019a","Type":"ContainerDied","Data":"838b02edd9f2504a916416325dbcc1514e4ff5d27ccb07f437321716eadc4ec6"} Nov 02 18:11:53.459532 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:53.459144 4471 scope.go:122] "RemoveContainer" containerID="dd1da4ad3c90b90be0ec428ffbd80ac1faffbdaca8fedfc44005142de19bc009" Nov 02 18:11:53.461000 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:53.460976 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9b8d5bbf-xnjnp" Nov 02 18:11:53.461126 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:53.461051 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9b8d5bbf-xnjnp" event={"ID":"83085553-108f-4d34-b8de-9d0c6e0b13e5","Type":"ContainerDied","Data":"e39d9b01f015e04fece5789da1105cc03e18cb9dd103de8b65c87aef2faaaf19"} Nov 02 18:11:53.463274 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:53.463247 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" event={"ID":"1aa74f62-fe2f-4238-9f58-4a0a20f7892a","Type":"ContainerDied","Data":"598d8ee7d0693ef2f0eccd864f5851fb1491644ace2eb4444e12c46c2a246aab"} Nov 02 18:11:53.463432 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:53.463359 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw" Nov 02 18:11:53.474122 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:53.474092 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" event={"ID":"d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5","Type":"ContainerDied","Data":"797b0b336abf3e9668cd62a660c5f82a3cd86a3689a224697714a5ee3ff31d37"} Nov 02 18:11:53.474347 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:53.474192 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk" Nov 02 18:11:53.478869 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:53.477504 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" event={"ID":"9ebaee6a-d3f4-4f0d-8485-6dd565c1e078","Type":"ContainerDied","Data":"62120c3ffe86bce5e25b609d63c46bcf43f032a9f95030dec194f2125b583ad8"} Nov 02 18:11:53.478869 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:53.477606 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc" Nov 02 18:11:53.480692 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:53.480669 4471 scope.go:122] "RemoveContainer" containerID="773191a50fdcf4ae1309a440e1737b7705be93ba616179653ccf5cb035fd7e22" Nov 02 18:11:53.497537 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:53.497515 4471 scope.go:122] "RemoveContainer" containerID="23a9aaf637c69775424d91c36553ff299b37381bff46db3b401612c0853ec643" Nov 02 18:11:53.502024 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:53.501996 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p" Nov 02 18:11:53.512025 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:53.512003 4471 scope.go:122] "RemoveContainer" containerID="032035c86e70ca74d40491c92948cb2032b9912590738e4442690c6427481490" Nov 02 18:11:53.524496 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:53.524456 4471 scope.go:122] "RemoveContainer" containerID="3bb3b7e01e995024c94215d5661cd6350bed5f90ed1c81bb2169c1a1544fdc66" Nov 02 18:11:53.556234 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:53.556189 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46"] Nov 02 18:11:53.566778 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:53.566739 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k8n46"] Nov 02 18:11:53.620481 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:53.620440 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc"] Nov 02 18:11:53.628050 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:53.628018 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-n75gc"] Nov 02 18:11:53.688654 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:53.688617 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9b8d5bbf-xnjnp"] Nov 02 18:11:53.705677 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:53.705625 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9b8d5bbf-xnjnp"] Nov 02 18:11:53.892686 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:53.892650 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw"] Nov 02 18:11:53.895614 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:53.895581 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7l8nw"] Nov 02 18:11:53.923496 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:53.923431 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk"] Nov 02 18:11:53.932102 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:53.932072 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gg9fk"] Nov 02 18:11:54.400833 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:54.400798 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a209f-789b5bc59f-lgg4z" podUID="dd2eb7cb-d5ff-4351-84ac-f19fabf7e616" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:11:54.442554 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:54.442520 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aa74f62-fe2f-4238-9f58-4a0a20f7892a" path="/var/lib/kubelet/pods/1aa74f62-fe2f-4238-9f58-4a0a20f7892a/volumes" Nov 02 18:11:54.443092 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:54.443077 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c047abd-2c48-4b50-ba99-579d0f2cc3fd" path="/var/lib/kubelet/pods/2c047abd-2c48-4b50-ba99-579d0f2cc3fd/volumes" Nov 02 18:11:54.443585 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:54.443573 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83085553-108f-4d34-b8de-9d0c6e0b13e5" path="/var/lib/kubelet/pods/83085553-108f-4d34-b8de-9d0c6e0b13e5/volumes" Nov 02 18:11:54.444461 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:54.444445 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ebaee6a-d3f4-4f0d-8485-6dd565c1e078" path="/var/lib/kubelet/pods/9ebaee6a-d3f4-4f0d-8485-6dd565c1e078/volumes" Nov 02 18:11:54.444905 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:54.444892 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd908c8a-4026-4da6-95c5-34112ecc019a" path="/var/lib/kubelet/pods/cd908c8a-4026-4da6-95c5-34112ecc019a/volumes" Nov 02 18:11:54.445355 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:54.445339 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5" path="/var/lib/kubelet/pods/d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5/volumes" Nov 02 18:11:56.314178 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:56.314137 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b91583-66bb7b5959-5596q" podUID="d94a6978-2092-4c1d-8731-f18298e5369b" containerName="payload-pre-processing" containerID="cri-o://f90597617984abc94cdd86d1496200ac9b3df4ad6e241b3bde6127aa9dd3ebc8" gracePeriod=25 Nov 02 18:11:56.524906 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:56.524665 4471 generic.go:356] "Generic (PLEG): container finished" podID="d94a6978-2092-4c1d-8731-f18298e5369b" containerID="f90597617984abc94cdd86d1496200ac9b3df4ad6e241b3bde6127aa9dd3ebc8" exitCode=0 Nov 02 18:11:56.524906 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:56.524746 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b91583-66bb7b5959-5596q" event={"ID":"d94a6978-2092-4c1d-8731-f18298e5369b","Type":"ContainerDied","Data":"f90597617984abc94cdd86d1496200ac9b3df4ad6e241b3bde6127aa9dd3ebc8"} Nov 02 18:11:56.740547 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:56.740521 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b91583-66bb7b5959-5596q" Nov 02 18:11:56.747797 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:56.747773 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d94a6978-2092-4c1d-8731-f18298e5369b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d94a6978-2092-4c1d-8731-f18298e5369b-plugins-config-volume\") pod \"d94a6978-2092-4c1d-8731-f18298e5369b\" (UID: \"d94a6978-2092-4c1d-8731-f18298e5369b\") " Nov 02 18:11:56.748060 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:56.748044 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d94a6978-2092-4c1d-8731-f18298e5369b-kube-api-access-lfhsw\" (UniqueName: \"kubernetes.io/projected/d94a6978-2092-4c1d-8731-f18298e5369b-kube-api-access-lfhsw\") pod \"d94a6978-2092-4c1d-8731-f18298e5369b\" (UID: \"d94a6978-2092-4c1d-8731-f18298e5369b\") " Nov 02 18:11:56.748127 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:56.748105 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d94a6978-2092-4c1d-8731-f18298e5369b-plugins-config-volume" pod "d94a6978-2092-4c1d-8731-f18298e5369b" (UID: "d94a6978-2092-4c1d-8731-f18298e5369b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:56.748477 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:56.748454 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d94a6978-2092-4c1d-8731-f18298e5369b-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:56.750068 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:56.750042 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d94a6978-2092-4c1d-8731-f18298e5369b-kube-api-access-lfhsw" pod "d94a6978-2092-4c1d-8731-f18298e5369b" (UID: "d94a6978-2092-4c1d-8731-f18298e5369b"). InnerVolumeSpecName "kube-api-access-lfhsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:56.849698 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:56.849662 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfhsw\" (UniqueName: \"kubernetes.io/projected/d94a6978-2092-4c1d-8731-f18298e5369b-kube-api-access-lfhsw\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:57.534078 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:57.534043 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b91583-66bb7b5959-5596q" Nov 02 18:11:57.534548 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:57.534077 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b91583-66bb7b5959-5596q" event={"ID":"d94a6978-2092-4c1d-8731-f18298e5369b","Type":"ContainerDied","Data":"3400bccd90e613db0c114d623b4671722f4470e524aaf78489949985cf82cc6f"} Nov 02 18:11:57.534548 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:57.534147 4471 scope.go:122] "RemoveContainer" containerID="f90597617984abc94cdd86d1496200ac9b3df4ad6e241b3bde6127aa9dd3ebc8" Nov 02 18:11:57.590762 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:57.590720 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b91583-66bb7b5959-5596q"] Nov 02 18:11:57.603358 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:57.603292 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b91583-66bb7b5959-5596q"] Nov 02 18:11:58.447818 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.447788 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:11:58.450554 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.450525 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d94a6978-2092-4c1d-8731-f18298e5369b" path="/var/lib/kubelet/pods/d94a6978-2092-4c1d-8731-f18298e5369b/volumes" Nov 02 18:11:58.506619 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.506575 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf"] Nov 02 18:11:58.534658 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.534618 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74ec749a-738f-4509-a7e5-be6eb97f4777-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ec749a-738f-4509-a7e5-be6eb97f4777-istiod-ca-cert\") pod \"74ec749a-738f-4509-a7e5-be6eb97f4777\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " Nov 02 18:11:58.535453 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.534693 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74ec749a-738f-4509-a7e5-be6eb97f4777-kube-api-access-k8xkw\" (UniqueName: \"kubernetes.io/projected/74ec749a-738f-4509-a7e5-be6eb97f4777-kube-api-access-k8xkw\") pod \"74ec749a-738f-4509-a7e5-be6eb97f4777\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " Nov 02 18:11:58.535453 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.534788 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-credential-socket\") pod \"74ec749a-738f-4509-a7e5-be6eb97f4777\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " Nov 02 18:11:58.535453 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.534838 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-istio-data\") pod \"74ec749a-738f-4509-a7e5-be6eb97f4777\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " Nov 02 18:11:58.535453 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.534874 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74ec749a-738f-4509-a7e5-be6eb97f4777-istio-token\" (UniqueName: \"kubernetes.io/projected/74ec749a-738f-4509-a7e5-be6eb97f4777-istio-token\") pod \"74ec749a-738f-4509-a7e5-be6eb97f4777\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " Nov 02 18:11:58.535453 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.534955 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-workload-socket\") pod \"74ec749a-738f-4509-a7e5-be6eb97f4777\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " Nov 02 18:11:58.535453 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.534988 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74ec749a-738f-4509-a7e5-be6eb97f4777-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ec749a-738f-4509-a7e5-be6eb97f4777-istio-podinfo\") pod \"74ec749a-738f-4509-a7e5-be6eb97f4777\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " Nov 02 18:11:58.535453 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.535032 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-workload-certs\") pod \"74ec749a-738f-4509-a7e5-be6eb97f4777\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " Nov 02 18:11:58.535453 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.535070 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-istio-envoy\") pod \"74ec749a-738f-4509-a7e5-be6eb97f4777\" (UID: \"74ec749a-738f-4509-a7e5-be6eb97f4777\") " Nov 02 18:11:58.536237 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.536209 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74ec749a-738f-4509-a7e5-be6eb97f4777-istiod-ca-cert" pod "74ec749a-738f-4509-a7e5-be6eb97f4777" (UID: "74ec749a-738f-4509-a7e5-be6eb97f4777"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:58.536729 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.536702 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745"] Nov 02 18:11:58.540886 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.537820 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-istio-data" pod "74ec749a-738f-4509-a7e5-be6eb97f4777" (UID: "74ec749a-738f-4509-a7e5-be6eb97f4777"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:58.540886 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.538156 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-credential-socket" pod "74ec749a-738f-4509-a7e5-be6eb97f4777" (UID: "74ec749a-738f-4509-a7e5-be6eb97f4777"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:58.540886 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.538379 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-workload-socket" pod "74ec749a-738f-4509-a7e5-be6eb97f4777" (UID: "74ec749a-738f-4509-a7e5-be6eb97f4777"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:58.540886 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.538490 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-workload-certs" pod "74ec749a-738f-4509-a7e5-be6eb97f4777" (UID: "74ec749a-738f-4509-a7e5-be6eb97f4777"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:58.540886 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.539897 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-istio-envoy" pod "74ec749a-738f-4509-a7e5-be6eb97f4777" (UID: "74ec749a-738f-4509-a7e5-be6eb97f4777"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:58.540886 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.540850 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ec749a-738f-4509-a7e5-be6eb97f4777-kube-api-access-k8xkw" pod "74ec749a-738f-4509-a7e5-be6eb97f4777" (UID: "74ec749a-738f-4509-a7e5-be6eb97f4777"). InnerVolumeSpecName "kube-api-access-k8xkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:58.549422 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.549398 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74ec749a-738f-4509-a7e5-be6eb97f4777-istio-podinfo" pod "74ec749a-738f-4509-a7e5-be6eb97f4777" (UID: "74ec749a-738f-4509-a7e5-be6eb97f4777"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:11:58.550216 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.550189 4471 generic.go:356] "Generic (PLEG): container finished" podID="74ec749a-738f-4509-a7e5-be6eb97f4777" containerID="3a8171f7433846b9317402622b74a5b56ead44961ed13c1ef3f5bef281544230" exitCode=0 Nov 02 18:11:58.550364 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.550249 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" Nov 02 18:11:58.550364 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.550269 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" event={"ID":"74ec749a-738f-4509-a7e5-be6eb97f4777","Type":"ContainerDied","Data":"3a8171f7433846b9317402622b74a5b56ead44961ed13c1ef3f5bef281544230"} Nov 02 18:11:58.550501 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.550366 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr" event={"ID":"74ec749a-738f-4509-a7e5-be6eb97f4777","Type":"ContainerDied","Data":"3dc1ae34312fd0b8c8f712471644f41672cf659b5ca92e324417c61dcad0d449"} Nov 02 18:11:58.550501 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.550401 4471 scope.go:122] "RemoveContainer" containerID="3a8171f7433846b9317402622b74a5b56ead44961ed13c1ef3f5bef281544230" Nov 02 18:11:58.552749 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.552728 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ec749a-738f-4509-a7e5-be6eb97f4777-istio-token" pod "74ec749a-738f-4509-a7e5-be6eb97f4777" (UID: "74ec749a-738f-4509-a7e5-be6eb97f4777"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:58.575517 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.574535 4471 scope.go:122] "RemoveContainer" containerID="3a8171f7433846b9317402622b74a5b56ead44961ed13c1ef3f5bef281544230" Nov 02 18:11:58.589400 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:11:58.582047 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a8171f7433846b9317402622b74a5b56ead44961ed13c1ef3f5bef281544230\": container with ID starting with 3a8171f7433846b9317402622b74a5b56ead44961ed13c1ef3f5bef281544230 not found: ID does not exist" containerID="3a8171f7433846b9317402622b74a5b56ead44961ed13c1ef3f5bef281544230" Nov 02 18:11:58.589400 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.582095 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a8171f7433846b9317402622b74a5b56ead44961ed13c1ef3f5bef281544230"} err="failed to get container status \"3a8171f7433846b9317402622b74a5b56ead44961ed13c1ef3f5bef281544230\": rpc error: code = NotFound desc = could not find container \"3a8171f7433846b9317402622b74a5b56ead44961ed13c1ef3f5bef281544230\": container with ID starting with 3a8171f7433846b9317402622b74a5b56ead44961ed13c1ef3f5bef281544230 not found: ID does not exist" Nov 02 18:11:58.647731 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.637690 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:58.647731 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.637730 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ec749a-738f-4509-a7e5-be6eb97f4777-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:58.647731 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.637746 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:58.647731 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.637772 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:58.647731 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.637787 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ec749a-738f-4509-a7e5-be6eb97f4777-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:58.647731 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.637801 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8xkw\" (UniqueName: \"kubernetes.io/projected/74ec749a-738f-4509-a7e5-be6eb97f4777-kube-api-access-k8xkw\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:58.647731 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.637819 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:58.647731 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.637839 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ec749a-738f-4509-a7e5-be6eb97f4777-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:58.647731 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.637854 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ec749a-738f-4509-a7e5-be6eb97f4777-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:11:58.912761 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.912724 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr"] Nov 02 18:11:58.918525 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:11:58.918494 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5zcjr"] Nov 02 18:12:00.443204 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:00.443167 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74ec749a-738f-4509-a7e5-be6eb97f4777" path="/var/lib/kubelet/pods/74ec749a-738f-4509-a7e5-be6eb97f4777/volumes" Nov 02 18:12:00.462778 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:00.462737 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" containerID="cri-o://7c3a2c8754db1a5d87cd2e50329239a0a6d6c02106e58a0a6f23d52e6c160f9b" gracePeriod=45 Nov 02 18:12:00.800530 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:00.800502 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 18:12:00.882087 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:00.882016 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-model-cache\") pod \"6ce077ac-aaca-4afc-8090-2fb935b4a3e9\" (UID: \"6ce077ac-aaca-4afc-8090-2fb935b4a3e9\") " Nov 02 18:12:00.882272 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:00.882096 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-kube-api-access-58p85\" (UniqueName: \"kubernetes.io/projected/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-kube-api-access-58p85\") pod \"6ce077ac-aaca-4afc-8090-2fb935b4a3e9\" (UID: \"6ce077ac-aaca-4afc-8090-2fb935b4a3e9\") " Nov 02 18:12:00.882272 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:00.882159 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-tmp-dir\") pod \"6ce077ac-aaca-4afc-8090-2fb935b4a3e9\" (UID: \"6ce077ac-aaca-4afc-8090-2fb935b4a3e9\") " Nov 02 18:12:00.882272 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:00.882221 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-home\" (UniqueName: \"kubernetes.io/empty-dir/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-home\") pod \"6ce077ac-aaca-4afc-8090-2fb935b4a3e9\" (UID: \"6ce077ac-aaca-4afc-8090-2fb935b4a3e9\") " Nov 02 18:12:00.882502 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:00.882372 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-tls-certs\") pod \"6ce077ac-aaca-4afc-8090-2fb935b4a3e9\" (UID: \"6ce077ac-aaca-4afc-8090-2fb935b4a3e9\") " Nov 02 18:12:00.882502 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:00.882497 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-dshm\") pod \"6ce077ac-aaca-4afc-8090-2fb935b4a3e9\" (UID: \"6ce077ac-aaca-4afc-8090-2fb935b4a3e9\") " Nov 02 18:12:00.883891 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:00.882930 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-model-cache" pod "6ce077ac-aaca-4afc-8090-2fb935b4a3e9" (UID: "6ce077ac-aaca-4afc-8090-2fb935b4a3e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:00.883891 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:00.883216 4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-model-cache\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:00.883891 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:00.883573 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-home" pod "6ce077ac-aaca-4afc-8090-2fb935b4a3e9" (UID: "6ce077ac-aaca-4afc-8090-2fb935b4a3e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:00.883891 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:00.883831 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-tmp-dir" pod "6ce077ac-aaca-4afc-8090-2fb935b4a3e9" (UID: "6ce077ac-aaca-4afc-8090-2fb935b4a3e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:00.889397 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:00.889360 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-dshm" pod "6ce077ac-aaca-4afc-8090-2fb935b4a3e9" (UID: "6ce077ac-aaca-4afc-8090-2fb935b4a3e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:00.889541 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:00.889452 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-kube-api-access-58p85" pod "6ce077ac-aaca-4afc-8090-2fb935b4a3e9" (UID: "6ce077ac-aaca-4afc-8090-2fb935b4a3e9"). InnerVolumeSpecName "kube-api-access-58p85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:00.889689 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:00.889668 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-tls-certs" pod "6ce077ac-aaca-4afc-8090-2fb935b4a3e9" (UID: "6ce077ac-aaca-4afc-8090-2fb935b4a3e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:12:00.984565 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:00.984525 4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-tls-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:00.984565 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:00.984563 4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-dshm\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:00.984779 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:00.984578 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58p85\" (UniqueName: \"kubernetes.io/projected/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-kube-api-access-58p85\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:00.984779 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:00.984591 4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-tmp-dir\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:00.984779 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:00.984603 4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ce077ac-aaca-4afc-8090-2fb935b4a3e9-home\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:01.578275 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:01.578236 4471 generic.go:356] "Generic (PLEG): container finished" podID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerID="7c3a2c8754db1a5d87cd2e50329239a0a6d6c02106e58a0a6f23d52e6c160f9b" exitCode=0 Nov 02 18:12:01.578774 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:01.578293 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" Nov 02 18:12:01.578774 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:01.578349 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" event={"ID":"6ce077ac-aaca-4afc-8090-2fb935b4a3e9","Type":"ContainerDied","Data":"7c3a2c8754db1a5d87cd2e50329239a0a6d6c02106e58a0a6f23d52e6c160f9b"} Nov 02 18:12:01.578774 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:01.578394 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx" event={"ID":"6ce077ac-aaca-4afc-8090-2fb935b4a3e9","Type":"ContainerDied","Data":"3f54cc0face5f04e55d57a5b630e58aa8e2f1511cb86631180ec0e50a4912619"} Nov 02 18:12:01.578774 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:01.578411 4471 scope.go:122] "RemoveContainer" containerID="7c3a2c8754db1a5d87cd2e50329239a0a6d6c02106e58a0a6f23d52e6c160f9b" Nov 02 18:12:01.589292 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:01.589269 4471 scope.go:122] "RemoveContainer" containerID="7c3a2c8754db1a5d87cd2e50329239a0a6d6c02106e58a0a6f23d52e6c160f9b" Nov 02 18:12:01.589639 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:12:01.589612 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c3a2c8754db1a5d87cd2e50329239a0a6d6c02106e58a0a6f23d52e6c160f9b\": container with ID starting with 7c3a2c8754db1a5d87cd2e50329239a0a6d6c02106e58a0a6f23d52e6c160f9b not found: ID does not exist" containerID="7c3a2c8754db1a5d87cd2e50329239a0a6d6c02106e58a0a6f23d52e6c160f9b" Nov 02 18:12:01.589755 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:01.589648 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c3a2c8754db1a5d87cd2e50329239a0a6d6c02106e58a0a6f23d52e6c160f9b"} err="failed to get container status \"7c3a2c8754db1a5d87cd2e50329239a0a6d6c02106e58a0a6f23d52e6c160f9b\": rpc error: code = NotFound desc = could not find container \"7c3a2c8754db1a5d87cd2e50329239a0a6d6c02106e58a0a6f23d52e6c160f9b\": container with ID starting with 7c3a2c8754db1a5d87cd2e50329239a0a6d6c02106e58a0a6f23d52e6c160f9b not found: ID does not exist" Nov 02 18:12:01.632797 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:01.632755 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx"] Nov 02 18:12:01.645213 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:01.645178 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-120394/distinct-w1-120394-kserve-65cf4d8b55-sxlkx"] Nov 02 18:12:01.742457 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:01.742411 4471 patch_prober.go:29] interesting pod/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:12:01.742643 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:01.742473 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" podUID="b54fd98c-e8c7-4578-b027-5275834e8c72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:12:02.444911 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:02.444874 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" path="/var/lib/kubelet/pods/6ce077ac-aaca-4afc-8090-2fb935b4a3e9/volumes" Nov 02 18:12:03.597368 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:03.597291 4471 generic.go:356] "Generic (PLEG): container finished" podID="b54fd98c-e8c7-4578-b027-5275834e8c72" containerID="e515c5200f7cf6ac9ec59af8a997c85c4400ba615caac97d7a3d496945f2f677" exitCode=0 Nov 02 18:12:03.597368 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:03.597359 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" event={"ID":"b54fd98c-e8c7-4578-b027-5275834e8c72","Type":"ContainerDied","Data":"e515c5200f7cf6ac9ec59af8a997c85c4400ba615caac97d7a3d496945f2f677"} Nov 02 18:12:03.597866 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:03.597852 4471 scope.go:122] "RemoveContainer" containerID="e515c5200f7cf6ac9ec59af8a997c85c4400ba615caac97d7a3d496945f2f677" Nov 02 18:12:03.599731 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:03.599700 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:12:03.599837 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:03.599768 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:12:03.599837 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:03.599799 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:12:04.400554 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:04.400514 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a209f-789b5bc59f-lgg4z" podUID="dd2eb7cb-d5ff-4351-84ac-f19fabf7e616" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:12:04.613326 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:04.612996 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" event={"ID":"b54fd98c-e8c7-4578-b027-5275834e8c72","Type":"ContainerStarted","Data":"71b848ccedd2eb154590ba64e370d24213d32f1bbf81e3d2eee71e5c9e3773f0"} Nov 02 18:12:05.545035 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:05.543746 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:12:05.545035 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:05.543909 4471 patch_prober.go:29] interesting pod/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 02 18:12:05.545035 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:05.543976 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" podUID="b54fd98c-e8c7-4578-b027-5275834e8c72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 02 18:12:05.809212 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:05.809121 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s"] Nov 02 18:12:05.810031 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:05.809992 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" podUID="5a819a2f-01ae-45ba-8c60-f3382617c58c" containerName="istio-proxy" containerID="cri-o://d996ec065a24dfcd958bc124245dbf1713a865f0934908b38dbb7be195e0bbbc" gracePeriod=30 Nov 02 18:12:05.883013 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:05.882971 4471 patch_prober.go:29] interesting pod/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:12:05.883208 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:05.883051 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" podUID="5a819a2f-01ae-45ba-8c60-f3382617c58c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:12:06.530086 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:06.530051 4471 patch_prober.go:29] interesting pod/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 02 18:12:06.530350 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:06.530112 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" podUID="b54fd98c-e8c7-4578-b027-5275834e8c72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 02 18:12:06.935986 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:06.935887 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5"] Nov 02 18:12:07.529810 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:07.529778 4471 patch_prober.go:29] interesting pod/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 02 18:12:07.530013 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:07.529829 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" podUID="b54fd98c-e8c7-4578-b027-5275834e8c72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 02 18:12:08.529639 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:08.529603 4471 patch_prober.go:29] interesting pod/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 02 18:12:08.530040 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:08.529653 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" podUID="b54fd98c-e8c7-4578-b027-5275834e8c72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 02 18:12:09.203485 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:12:09.203437 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:12:09.529463 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:09.529429 4471 patch_prober.go:29] interesting pod/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 02 18:12:09.529653 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:09.529482 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" podUID="b54fd98c-e8c7-4578-b027-5275834e8c72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 02 18:12:09.667238 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:09.667200 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-504de7-786496d969-w8f9z"] Nov 02 18:12:09.955206 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:09.955119 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8"] Nov 02 18:12:10.386775 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:10.373248 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a209f-789b5bc59f-lgg4z" Nov 02 18:12:10.468324 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:10.455881 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd2eb7cb-d5ff-4351-84ac-f19fabf7e616-kube-api-access-8kvc7\" (UniqueName: \"kubernetes.io/projected/dd2eb7cb-d5ff-4351-84ac-f19fabf7e616-kube-api-access-8kvc7\") pod \"dd2eb7cb-d5ff-4351-84ac-f19fabf7e616\" (UID: \"dd2eb7cb-d5ff-4351-84ac-f19fabf7e616\") " Nov 02 18:12:10.468324 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:10.455955 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd2eb7cb-d5ff-4351-84ac-f19fabf7e616-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2eb7cb-d5ff-4351-84ac-f19fabf7e616-plugins-config-volume\") pod \"dd2eb7cb-d5ff-4351-84ac-f19fabf7e616\" (UID: \"dd2eb7cb-d5ff-4351-84ac-f19fabf7e616\") " Nov 02 18:12:10.468324 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:10.457138 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd2eb7cb-d5ff-4351-84ac-f19fabf7e616-plugins-config-volume" pod "dd2eb7cb-d5ff-4351-84ac-f19fabf7e616" (UID: "dd2eb7cb-d5ff-4351-84ac-f19fabf7e616"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:10.468324 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:10.463613 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd2eb7cb-d5ff-4351-84ac-f19fabf7e616-kube-api-access-8kvc7" pod "dd2eb7cb-d5ff-4351-84ac-f19fabf7e616" (UID: "dd2eb7cb-d5ff-4351-84ac-f19fabf7e616"). InnerVolumeSpecName "kube-api-access-8kvc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:10.534004 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:10.533973 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:12:10.534435 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:10.534414 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:12:10.535210 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:10.535192 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:12:10.558134 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:10.558096 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kvc7\" (UniqueName: \"kubernetes.io/projected/dd2eb7cb-d5ff-4351-84ac-f19fabf7e616-kube-api-access-8kvc7\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:10.558134 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:10.558125 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2eb7cb-d5ff-4351-84ac-f19fabf7e616-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:10.689094 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:10.689056 4471 generic.go:356] "Generic (PLEG): container finished" podID="dd2eb7cb-d5ff-4351-84ac-f19fabf7e616" containerID="47345402a5b9cb8bb2b10d5ba64e407fc614e603dc190b7b317dbefb51ca51a6" exitCode=0 Nov 02 18:12:10.689620 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:10.689125 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a209f-789b5bc59f-lgg4z" Nov 02 18:12:10.689620 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:10.689584 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a209f-789b5bc59f-lgg4z" event={"ID":"dd2eb7cb-d5ff-4351-84ac-f19fabf7e616","Type":"ContainerDied","Data":"47345402a5b9cb8bb2b10d5ba64e407fc614e603dc190b7b317dbefb51ca51a6"} Nov 02 18:12:10.689745 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:10.689658 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a209f-789b5bc59f-lgg4z" event={"ID":"dd2eb7cb-d5ff-4351-84ac-f19fabf7e616","Type":"ContainerDied","Data":"05c48c7ddda25695f422d937e1bb97394dc591cc69333d45cc7c1ad8d43ec25f"} Nov 02 18:12:10.689745 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:10.689687 4471 scope.go:122] "RemoveContainer" containerID="47345402a5b9cb8bb2b10d5ba64e407fc614e603dc190b7b317dbefb51ca51a6" Nov 02 18:12:10.707186 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:10.707161 4471 scope.go:122] "RemoveContainer" containerID="47345402a5b9cb8bb2b10d5ba64e407fc614e603dc190b7b317dbefb51ca51a6" Nov 02 18:12:10.708346 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:12:10.708045 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47345402a5b9cb8bb2b10d5ba64e407fc614e603dc190b7b317dbefb51ca51a6\": container with ID starting with 47345402a5b9cb8bb2b10d5ba64e407fc614e603dc190b7b317dbefb51ca51a6 not found: ID does not exist" containerID="47345402a5b9cb8bb2b10d5ba64e407fc614e603dc190b7b317dbefb51ca51a6" Nov 02 18:12:10.708346 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:10.708082 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47345402a5b9cb8bb2b10d5ba64e407fc614e603dc190b7b317dbefb51ca51a6"} err="failed to get container status \"47345402a5b9cb8bb2b10d5ba64e407fc614e603dc190b7b317dbefb51ca51a6\": rpc error: code = NotFound desc = could not find container \"47345402a5b9cb8bb2b10d5ba64e407fc614e603dc190b7b317dbefb51ca51a6\": container with ID starting with 47345402a5b9cb8bb2b10d5ba64e407fc614e603dc190b7b317dbefb51ca51a6 not found: ID does not exist" Nov 02 18:12:10.751990 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:10.751952 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8a209f-789b5bc59f-lgg4z"] Nov 02 18:12:10.755194 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:10.755167 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8a209f-789b5bc59f-lgg4z"] Nov 02 18:12:11.089228 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.089160 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 18:12:11.167941 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.167903 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-workload-certs\") pod \"5a819a2f-01ae-45ba-8c60-f3382617c58c\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " Nov 02 18:12:11.168151 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.168029 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-credential-socket\") pod \"5a819a2f-01ae-45ba-8c60-f3382617c58c\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " Nov 02 18:12:11.168151 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.168062 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-workload-socket\") pod \"5a819a2f-01ae-45ba-8c60-f3382617c58c\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " Nov 02 18:12:11.168287 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.168202 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-workload-certs" pod "5a819a2f-01ae-45ba-8c60-f3382617c58c" (UID: "5a819a2f-01ae-45ba-8c60-f3382617c58c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:11.168287 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.168205 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a819a2f-01ae-45ba-8c60-f3382617c58c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a819a2f-01ae-45ba-8c60-f3382617c58c-istiod-ca-cert\") pod \"5a819a2f-01ae-45ba-8c60-f3382617c58c\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " Nov 02 18:12:11.168419 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.168296 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-istio-envoy\") pod \"5a819a2f-01ae-45ba-8c60-f3382617c58c\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " Nov 02 18:12:11.168483 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.168422 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-credential-socket" pod "5a819a2f-01ae-45ba-8c60-f3382617c58c" (UID: "5a819a2f-01ae-45ba-8c60-f3382617c58c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:11.168483 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.168438 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-workload-socket" pod "5a819a2f-01ae-45ba-8c60-f3382617c58c" (UID: "5a819a2f-01ae-45ba-8c60-f3382617c58c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:11.168556 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.168455 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a819a2f-01ae-45ba-8c60-f3382617c58c-kube-api-access-t2zdz\" (UniqueName: \"kubernetes.io/projected/5a819a2f-01ae-45ba-8c60-f3382617c58c-kube-api-access-t2zdz\") pod \"5a819a2f-01ae-45ba-8c60-f3382617c58c\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " Nov 02 18:12:11.168600 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.168575 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-istio-data\") pod \"5a819a2f-01ae-45ba-8c60-f3382617c58c\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " Nov 02 18:12:11.168664 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.168627 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a819a2f-01ae-45ba-8c60-f3382617c58c-istiod-ca-cert" pod "5a819a2f-01ae-45ba-8c60-f3382617c58c" (UID: "5a819a2f-01ae-45ba-8c60-f3382617c58c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:11.168729 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.168666 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a819a2f-01ae-45ba-8c60-f3382617c58c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a819a2f-01ae-45ba-8c60-f3382617c58c-istio-podinfo\") pod \"5a819a2f-01ae-45ba-8c60-f3382617c58c\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " Nov 02 18:12:11.168729 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.168694 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a819a2f-01ae-45ba-8c60-f3382617c58c-istio-token\" (UniqueName: \"kubernetes.io/projected/5a819a2f-01ae-45ba-8c60-f3382617c58c-istio-token\") pod \"5a819a2f-01ae-45ba-8c60-f3382617c58c\" (UID: \"5a819a2f-01ae-45ba-8c60-f3382617c58c\") " Nov 02 18:12:11.168875 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.168842 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-istio-data" pod "5a819a2f-01ae-45ba-8c60-f3382617c58c" (UID: "5a819a2f-01ae-45ba-8c60-f3382617c58c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:11.169297 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.169275 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:11.169297 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.169322 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:11.169521 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.169355 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:11.169521 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.169373 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:11.169521 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.169386 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a819a2f-01ae-45ba-8c60-f3382617c58c-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:11.171022 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.170997 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-istio-envoy" pod "5a819a2f-01ae-45ba-8c60-f3382617c58c" (UID: "5a819a2f-01ae-45ba-8c60-f3382617c58c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:11.171166 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.171025 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a819a2f-01ae-45ba-8c60-f3382617c58c-kube-api-access-t2zdz" pod "5a819a2f-01ae-45ba-8c60-f3382617c58c" (UID: "5a819a2f-01ae-45ba-8c60-f3382617c58c"). InnerVolumeSpecName "kube-api-access-t2zdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:11.171166 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.171026 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a819a2f-01ae-45ba-8c60-f3382617c58c-istio-podinfo" pod "5a819a2f-01ae-45ba-8c60-f3382617c58c" (UID: "5a819a2f-01ae-45ba-8c60-f3382617c58c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:12:11.171339 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.171203 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a819a2f-01ae-45ba-8c60-f3382617c58c-istio-token" pod "5a819a2f-01ae-45ba-8c60-f3382617c58c" (UID: "5a819a2f-01ae-45ba-8c60-f3382617c58c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:11.271159 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.271120 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a819a2f-01ae-45ba-8c60-f3382617c58c-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:11.271159 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.271156 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2zdz\" (UniqueName: \"kubernetes.io/projected/5a819a2f-01ae-45ba-8c60-f3382617c58c-kube-api-access-t2zdz\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:11.271159 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.271166 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a819a2f-01ae-45ba-8c60-f3382617c58c-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:11.271452 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.271175 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a819a2f-01ae-45ba-8c60-f3382617c58c-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:11.701283 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.701244 4471 generic.go:356] "Generic (PLEG): container finished" podID="5a819a2f-01ae-45ba-8c60-f3382617c58c" containerID="d996ec065a24dfcd958bc124245dbf1713a865f0934908b38dbb7be195e0bbbc" exitCode=0 Nov 02 18:12:11.701732 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.701331 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" Nov 02 18:12:11.701732 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.701334 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" event={"ID":"5a819a2f-01ae-45ba-8c60-f3382617c58c","Type":"ContainerDied","Data":"d996ec065a24dfcd958bc124245dbf1713a865f0934908b38dbb7be195e0bbbc"} Nov 02 18:12:11.701732 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.701396 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s" event={"ID":"5a819a2f-01ae-45ba-8c60-f3382617c58c","Type":"ContainerDied","Data":"cbb8a64632149e2f4105fb5f3e49d1d7d0089d5f951baf0ba26536aba2adf86c"} Nov 02 18:12:11.701732 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.701427 4471 scope.go:122] "RemoveContainer" containerID="d996ec065a24dfcd958bc124245dbf1713a865f0934908b38dbb7be195e0bbbc" Nov 02 18:12:11.714740 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.714717 4471 scope.go:122] "RemoveContainer" containerID="ded9c25461a48a974f0c690ed9509607e191a2124ad1cd4bec1c3bea33eab9aa" Nov 02 18:12:11.724708 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.724681 4471 scope.go:122] "RemoveContainer" containerID="d996ec065a24dfcd958bc124245dbf1713a865f0934908b38dbb7be195e0bbbc" Nov 02 18:12:11.724972 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:12:11.724952 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d996ec065a24dfcd958bc124245dbf1713a865f0934908b38dbb7be195e0bbbc\": container with ID starting with d996ec065a24dfcd958bc124245dbf1713a865f0934908b38dbb7be195e0bbbc not found: ID does not exist" containerID="d996ec065a24dfcd958bc124245dbf1713a865f0934908b38dbb7be195e0bbbc" Nov 02 18:12:11.725037 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.724983 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d996ec065a24dfcd958bc124245dbf1713a865f0934908b38dbb7be195e0bbbc"} err="failed to get container status \"d996ec065a24dfcd958bc124245dbf1713a865f0934908b38dbb7be195e0bbbc\": rpc error: code = NotFound desc = could not find container \"d996ec065a24dfcd958bc124245dbf1713a865f0934908b38dbb7be195e0bbbc\": container with ID starting with d996ec065a24dfcd958bc124245dbf1713a865f0934908b38dbb7be195e0bbbc not found: ID does not exist" Nov 02 18:12:11.725037 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.725000 4471 scope.go:122] "RemoveContainer" containerID="ded9c25461a48a974f0c690ed9509607e191a2124ad1cd4bec1c3bea33eab9aa" Nov 02 18:12:11.725231 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:12:11.725214 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ded9c25461a48a974f0c690ed9509607e191a2124ad1cd4bec1c3bea33eab9aa\": container with ID starting with ded9c25461a48a974f0c690ed9509607e191a2124ad1cd4bec1c3bea33eab9aa not found: ID does not exist" containerID="ded9c25461a48a974f0c690ed9509607e191a2124ad1cd4bec1c3bea33eab9aa" Nov 02 18:12:11.725273 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.725236 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ded9c25461a48a974f0c690ed9509607e191a2124ad1cd4bec1c3bea33eab9aa"} err="failed to get container status \"ded9c25461a48a974f0c690ed9509607e191a2124ad1cd4bec1c3bea33eab9aa\": rpc error: code = NotFound desc = could not find container \"ded9c25461a48a974f0c690ed9509607e191a2124ad1cd4bec1c3bea33eab9aa\": container with ID starting with ded9c25461a48a974f0c690ed9509607e191a2124ad1cd4bec1c3bea33eab9aa not found: ID does not exist" Nov 02 18:12:11.782263 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.780364 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s"] Nov 02 18:12:11.786326 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:11.785500 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-9mt7s"] Nov 02 18:12:12.442395 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:12.442359 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a819a2f-01ae-45ba-8c60-f3382617c58c" path="/var/lib/kubelet/pods/5a819a2f-01ae-45ba-8c60-f3382617c58c/volumes" Nov 02 18:12:12.442843 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:12.442828 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd2eb7cb-d5ff-4351-84ac-f19fabf7e616" path="/var/lib/kubelet/pods/dd2eb7cb-d5ff-4351-84ac-f19fabf7e616/volumes" Nov 02 18:12:13.534585 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:13.534534 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" containerID="cri-o://10f6d0d2a3460de32ba19c0328b3e18836c632df28c44794e9d80b16b8800151" gracePeriod=45 Nov 02 18:12:13.607197 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:13.607161 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" containerID="cri-o://b8c969d3404243c9d12216bd17d9562b58249fcd9522d82e7a11ed8a2fdcf7d4" gracePeriod=45 Nov 02 18:12:13.721601 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:13.721562 4471 generic.go:356] "Generic (PLEG): container finished" podID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerID="10f6d0d2a3460de32ba19c0328b3e18836c632df28c44794e9d80b16b8800151" exitCode=0 Nov 02 18:12:13.721770 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:13.721596 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" event={"ID":"abaa1ee3-4cc9-47d4-9c44-0ce826b1e496","Type":"ContainerDied","Data":"10f6d0d2a3460de32ba19c0328b3e18836c632df28c44794e9d80b16b8800151"} Nov 02 18:12:13.919387 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:13.919352 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 18:12:13.939919 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:13.939894 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 18:12:14.012044 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.012014 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520fe9ff-603f-4df8-8b71-06ccfdd1019e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/520fe9ff-603f-4df8-8b71-06ccfdd1019e-tmp-dir\") pod \"520fe9ff-603f-4df8-8b71-06ccfdd1019e\" (UID: \"520fe9ff-603f-4df8-8b71-06ccfdd1019e\") " Nov 02 18:12:14.012263 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.012077 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/520fe9ff-603f-4df8-8b71-06ccfdd1019e-tls-certs\" (UniqueName: \"kubernetes.io/secret/520fe9ff-603f-4df8-8b71-06ccfdd1019e-tls-certs\") pod \"520fe9ff-603f-4df8-8b71-06ccfdd1019e\" (UID: \"520fe9ff-603f-4df8-8b71-06ccfdd1019e\") " Nov 02 18:12:14.012263 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.012127 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-home\" (UniqueName: \"kubernetes.io/empty-dir/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-home\") pod \"abaa1ee3-4cc9-47d4-9c44-0ce826b1e496\" (UID: \"abaa1ee3-4cc9-47d4-9c44-0ce826b1e496\") " Nov 02 18:12:14.012263 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.012176 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520fe9ff-603f-4df8-8b71-06ccfdd1019e-home\" (UniqueName: \"kubernetes.io/empty-dir/520fe9ff-603f-4df8-8b71-06ccfdd1019e-home\") pod \"520fe9ff-603f-4df8-8b71-06ccfdd1019e\" (UID: \"520fe9ff-603f-4df8-8b71-06ccfdd1019e\") " Nov 02 18:12:14.012464 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.012271 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-kube-api-access-zwnbz\" (UniqueName: \"kubernetes.io/projected/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-kube-api-access-zwnbz\") pod \"abaa1ee3-4cc9-47d4-9c44-0ce826b1e496\" (UID: \"abaa1ee3-4cc9-47d4-9c44-0ce826b1e496\") " Nov 02 18:12:14.012464 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.012352 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520fe9ff-603f-4df8-8b71-06ccfdd1019e-tmp-dir" pod "520fe9ff-603f-4df8-8b71-06ccfdd1019e" (UID: "520fe9ff-603f-4df8-8b71-06ccfdd1019e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:14.012464 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.012411 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-model-cache\") pod \"abaa1ee3-4cc9-47d4-9c44-0ce826b1e496\" (UID: \"abaa1ee3-4cc9-47d4-9c44-0ce826b1e496\") " Nov 02 18:12:14.012464 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.012442 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-tls-certs\" (UniqueName: \"kubernetes.io/secret/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-tls-certs\") pod \"abaa1ee3-4cc9-47d4-9c44-0ce826b1e496\" (UID: \"abaa1ee3-4cc9-47d4-9c44-0ce826b1e496\") " Nov 02 18:12:14.012695 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.012473 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/520fe9ff-603f-4df8-8b71-06ccfdd1019e-kube-api-access-xhk4z\" (UniqueName: \"kubernetes.io/projected/520fe9ff-603f-4df8-8b71-06ccfdd1019e-kube-api-access-xhk4z\") pod \"520fe9ff-603f-4df8-8b71-06ccfdd1019e\" (UID: \"520fe9ff-603f-4df8-8b71-06ccfdd1019e\") " Nov 02 18:12:14.012695 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.012525 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-dshm\" (UniqueName: \"kubernetes.io/empty-dir/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-dshm\") pod \"abaa1ee3-4cc9-47d4-9c44-0ce826b1e496\" (UID: \"abaa1ee3-4cc9-47d4-9c44-0ce826b1e496\") " Nov 02 18:12:14.012695 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.012534 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520fe9ff-603f-4df8-8b71-06ccfdd1019e-home" pod "520fe9ff-603f-4df8-8b71-06ccfdd1019e" (UID: "520fe9ff-603f-4df8-8b71-06ccfdd1019e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:14.012695 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.012631 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520fe9ff-603f-4df8-8b71-06ccfdd1019e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/520fe9ff-603f-4df8-8b71-06ccfdd1019e-model-cache\") pod \"520fe9ff-603f-4df8-8b71-06ccfdd1019e\" (UID: \"520fe9ff-603f-4df8-8b71-06ccfdd1019e\") " Nov 02 18:12:14.012695 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.012664 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520fe9ff-603f-4df8-8b71-06ccfdd1019e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/520fe9ff-603f-4df8-8b71-06ccfdd1019e-dshm\") pod \"520fe9ff-603f-4df8-8b71-06ccfdd1019e\" (UID: \"520fe9ff-603f-4df8-8b71-06ccfdd1019e\") " Nov 02 18:12:14.012961 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.012736 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-tmp-dir\") pod \"abaa1ee3-4cc9-47d4-9c44-0ce826b1e496\" (UID: \"abaa1ee3-4cc9-47d4-9c44-0ce826b1e496\") " Nov 02 18:12:14.014567 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.013423 4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/520fe9ff-603f-4df8-8b71-06ccfdd1019e-home\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:14.014567 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.013447 4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/520fe9ff-603f-4df8-8b71-06ccfdd1019e-tmp-dir\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:14.014567 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.013507 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520fe9ff-603f-4df8-8b71-06ccfdd1019e-model-cache" pod "520fe9ff-603f-4df8-8b71-06ccfdd1019e" (UID: "520fe9ff-603f-4df8-8b71-06ccfdd1019e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:14.014567 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.013777 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-tmp-dir" pod "abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" (UID: "abaa1ee3-4cc9-47d4-9c44-0ce826b1e496"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:14.014567 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.014170 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-model-cache" pod "abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" (UID: "abaa1ee3-4cc9-47d4-9c44-0ce826b1e496"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:14.015094 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.014847 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-home" pod "abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" (UID: "abaa1ee3-4cc9-47d4-9c44-0ce826b1e496"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:14.019961 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.019927 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/520fe9ff-603f-4df8-8b71-06ccfdd1019e-kube-api-access-xhk4z" pod "520fe9ff-603f-4df8-8b71-06ccfdd1019e" (UID: "520fe9ff-603f-4df8-8b71-06ccfdd1019e"). InnerVolumeSpecName "kube-api-access-xhk4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:14.027611 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.027586 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-tls-certs" pod "abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" (UID: "abaa1ee3-4cc9-47d4-9c44-0ce826b1e496"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:12:14.028715 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.028692 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-dshm" pod "abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" (UID: "abaa1ee3-4cc9-47d4-9c44-0ce826b1e496"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:14.028894 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.028868 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-kube-api-access-zwnbz" pod "abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" (UID: "abaa1ee3-4cc9-47d4-9c44-0ce826b1e496"). InnerVolumeSpecName "kube-api-access-zwnbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:14.030090 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.030062 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/520fe9ff-603f-4df8-8b71-06ccfdd1019e-tls-certs" pod "520fe9ff-603f-4df8-8b71-06ccfdd1019e" (UID: "520fe9ff-603f-4df8-8b71-06ccfdd1019e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:12:14.030207 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.030190 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520fe9ff-603f-4df8-8b71-06ccfdd1019e-dshm" pod "520fe9ff-603f-4df8-8b71-06ccfdd1019e" (UID: "520fe9ff-603f-4df8-8b71-06ccfdd1019e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:14.115065 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.114979 4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/520fe9ff-603f-4df8-8b71-06ccfdd1019e-tls-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:14.115065 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.115008 4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-home\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:14.115065 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.115020 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwnbz\" (UniqueName: \"kubernetes.io/projected/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-kube-api-access-zwnbz\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:14.115065 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.115034 4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-model-cache\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:14.115065 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.115045 4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-tls-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:14.115065 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.115053 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhk4z\" (UniqueName: \"kubernetes.io/projected/520fe9ff-603f-4df8-8b71-06ccfdd1019e-kube-api-access-xhk4z\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:14.115065 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.115063 4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-dshm\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:14.115065 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.115072 4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/520fe9ff-603f-4df8-8b71-06ccfdd1019e-model-cache\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:14.115473 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.115079 4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/520fe9ff-603f-4df8-8b71-06ccfdd1019e-dshm\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:14.115473 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.115088 4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496-tmp-dir\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:14.667877 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.667833 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-504de7-786496d969-w8f9z" podUID="a06f649d-baff-4ee4-b35b-bc5966df3067" containerName="payload-processing" containerID="cri-o://fe64d53bff10de18fe8ab223ec6ea4abacf9c938f07a88232c018222ce6746ff" gracePeriod=25 Nov 02 18:12:14.729758 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.729681 4471 generic.go:356] "Generic (PLEG): container finished" podID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerID="b8c969d3404243c9d12216bd17d9562b58249fcd9522d82e7a11ed8a2fdcf7d4" exitCode=0 Nov 02 18:12:14.729758 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.729724 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" Nov 02 18:12:14.729965 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.729761 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" event={"ID":"520fe9ff-603f-4df8-8b71-06ccfdd1019e","Type":"ContainerDied","Data":"b8c969d3404243c9d12216bd17d9562b58249fcd9522d82e7a11ed8a2fdcf7d4"} Nov 02 18:12:14.729965 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.729813 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745" event={"ID":"520fe9ff-603f-4df8-8b71-06ccfdd1019e","Type":"ContainerDied","Data":"76b4422da004c196cf173b84b9e3727964fb76f436906eb8fae4d34abbddeece"} Nov 02 18:12:14.729965 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.729836 4471 scope.go:122] "RemoveContainer" containerID="b8c969d3404243c9d12216bd17d9562b58249fcd9522d82e7a11ed8a2fdcf7d4" Nov 02 18:12:14.731350 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.731330 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" event={"ID":"abaa1ee3-4cc9-47d4-9c44-0ce826b1e496","Type":"ContainerDied","Data":"e3cd802b95c42af44ec62411e293ac47e84204f1662317821140c4a8c851fba0"} Nov 02 18:12:14.731350 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.731341 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf" Nov 02 18:12:14.779551 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.779532 4471 scope.go:122] "RemoveContainer" containerID="b8c969d3404243c9d12216bd17d9562b58249fcd9522d82e7a11ed8a2fdcf7d4" Nov 02 18:12:14.779895 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:12:14.779866 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8c969d3404243c9d12216bd17d9562b58249fcd9522d82e7a11ed8a2fdcf7d4\": container with ID starting with b8c969d3404243c9d12216bd17d9562b58249fcd9522d82e7a11ed8a2fdcf7d4 not found: ID does not exist" containerID="b8c969d3404243c9d12216bd17d9562b58249fcd9522d82e7a11ed8a2fdcf7d4" Nov 02 18:12:14.779996 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.779904 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8c969d3404243c9d12216bd17d9562b58249fcd9522d82e7a11ed8a2fdcf7d4"} err="failed to get container status \"b8c969d3404243c9d12216bd17d9562b58249fcd9522d82e7a11ed8a2fdcf7d4\": rpc error: code = NotFound desc = could not find container \"b8c969d3404243c9d12216bd17d9562b58249fcd9522d82e7a11ed8a2fdcf7d4\": container with ID starting with b8c969d3404243c9d12216bd17d9562b58249fcd9522d82e7a11ed8a2fdcf7d4 not found: ID does not exist" Nov 02 18:12:14.779996 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.779922 4471 scope.go:122] "RemoveContainer" containerID="10f6d0d2a3460de32ba19c0328b3e18836c632df28c44794e9d80b16b8800151" Nov 02 18:12:14.808632 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.808605 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745"] Nov 02 18:12:14.826922 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.826887 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-f0a3b8/simulator-w2-f0a3b8-kserve-56df57c5cc-7m745"] Nov 02 18:12:14.862719 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.862686 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf"] Nov 02 18:12:14.871145 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.871112 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-f0a3b8/distinct-2-w2-f0a3b8-kserve-75f7b48c64-tkpcf"] Nov 02 18:12:14.956783 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:14.956757 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-504de7-786496d969-w8f9z" Nov 02 18:12:15.025603 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:15.025573 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a06f649d-baff-4ee4-b35b-bc5966df3067-kube-api-access-4wn57\" (UniqueName: \"kubernetes.io/projected/a06f649d-baff-4ee4-b35b-bc5966df3067-kube-api-access-4wn57\") pod \"a06f649d-baff-4ee4-b35b-bc5966df3067\" (UID: \"a06f649d-baff-4ee4-b35b-bc5966df3067\") " Nov 02 18:12:15.025788 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:15.025712 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a06f649d-baff-4ee4-b35b-bc5966df3067-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a06f649d-baff-4ee4-b35b-bc5966df3067-plugins-config-volume\") pod \"a06f649d-baff-4ee4-b35b-bc5966df3067\" (UID: \"a06f649d-baff-4ee4-b35b-bc5966df3067\") " Nov 02 18:12:15.026116 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:15.026089 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a06f649d-baff-4ee4-b35b-bc5966df3067-plugins-config-volume" pod "a06f649d-baff-4ee4-b35b-bc5966df3067" (UID: "a06f649d-baff-4ee4-b35b-bc5966df3067"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:15.026358 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:15.026340 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a06f649d-baff-4ee4-b35b-bc5966df3067-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:15.027746 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:15.027714 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a06f649d-baff-4ee4-b35b-bc5966df3067-kube-api-access-4wn57" pod "a06f649d-baff-4ee4-b35b-bc5966df3067" (UID: "a06f649d-baff-4ee4-b35b-bc5966df3067"). InnerVolumeSpecName "kube-api-access-4wn57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:15.128127 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:15.128092 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wn57\" (UniqueName: \"kubernetes.io/projected/a06f649d-baff-4ee4-b35b-bc5966df3067-kube-api-access-4wn57\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:15.739787 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:15.739750 4471 generic.go:356] "Generic (PLEG): container finished" podID="a06f649d-baff-4ee4-b35b-bc5966df3067" containerID="fe64d53bff10de18fe8ab223ec6ea4abacf9c938f07a88232c018222ce6746ff" exitCode=0 Nov 02 18:12:15.740285 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:15.739817 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-504de7-786496d969-w8f9z" Nov 02 18:12:15.740285 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:15.739839 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-504de7-786496d969-w8f9z" event={"ID":"a06f649d-baff-4ee4-b35b-bc5966df3067","Type":"ContainerDied","Data":"fe64d53bff10de18fe8ab223ec6ea4abacf9c938f07a88232c018222ce6746ff"} Nov 02 18:12:15.740285 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:15.739909 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-504de7-786496d969-w8f9z" event={"ID":"a06f649d-baff-4ee4-b35b-bc5966df3067","Type":"ContainerDied","Data":"921673d410156d00ae64cd04a313b2ca8b514f3c38c33c999af90778604cf5d5"} Nov 02 18:12:15.740285 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:15.739925 4471 scope.go:122] "RemoveContainer" containerID="fe64d53bff10de18fe8ab223ec6ea4abacf9c938f07a88232c018222ce6746ff" Nov 02 18:12:15.750914 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:15.750893 4471 scope.go:122] "RemoveContainer" containerID="fe64d53bff10de18fe8ab223ec6ea4abacf9c938f07a88232c018222ce6746ff" Nov 02 18:12:15.751196 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:12:15.751176 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe64d53bff10de18fe8ab223ec6ea4abacf9c938f07a88232c018222ce6746ff\": container with ID starting with fe64d53bff10de18fe8ab223ec6ea4abacf9c938f07a88232c018222ce6746ff not found: ID does not exist" containerID="fe64d53bff10de18fe8ab223ec6ea4abacf9c938f07a88232c018222ce6746ff" Nov 02 18:12:15.751272 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:15.751202 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe64d53bff10de18fe8ab223ec6ea4abacf9c938f07a88232c018222ce6746ff"} err="failed to get container status \"fe64d53bff10de18fe8ab223ec6ea4abacf9c938f07a88232c018222ce6746ff\": rpc error: code = NotFound desc = could not find container \"fe64d53bff10de18fe8ab223ec6ea4abacf9c938f07a88232c018222ce6746ff\": container with ID starting with fe64d53bff10de18fe8ab223ec6ea4abacf9c938f07a88232c018222ce6746ff not found: ID does not exist" Nov 02 18:12:15.794564 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:15.794532 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-504de7-786496d969-w8f9z"] Nov 02 18:12:15.806889 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:15.806864 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-504de7-786496d969-w8f9z"] Nov 02 18:12:16.451444 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:16.451410 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" path="/var/lib/kubelet/pods/520fe9ff-603f-4df8-8b71-06ccfdd1019e/volumes" Nov 02 18:12:16.451910 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:16.451891 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a06f649d-baff-4ee4-b35b-bc5966df3067" path="/var/lib/kubelet/pods/a06f649d-baff-4ee4-b35b-bc5966df3067/volumes" Nov 02 18:12:16.452358 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:16.452343 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" path="/var/lib/kubelet/pods/abaa1ee3-4cc9-47d4-9c44-0ce826b1e496/volumes" Nov 02 18:12:21.987301 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:21.987257 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5" podUID="abbd398c-ad2f-4d13-91d0-fbcf97802571" containerName="main" containerID="cri-o://721d153d11b375908cdfa847071e43d44cb8544303522bc1fca5a8349f46731b" gracePeriod=45 Nov 02 18:12:22.271658 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:22.271634 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5" Nov 02 18:12:22.436237 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:22.436205 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abbd398c-ad2f-4d13-91d0-fbcf97802571-home\" (UniqueName: \"kubernetes.io/empty-dir/abbd398c-ad2f-4d13-91d0-fbcf97802571-home\") pod \"abbd398c-ad2f-4d13-91d0-fbcf97802571\" (UID: \"abbd398c-ad2f-4d13-91d0-fbcf97802571\") " Nov 02 18:12:22.436438 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:22.436362 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abbd398c-ad2f-4d13-91d0-fbcf97802571-dshm\" (UniqueName: \"kubernetes.io/empty-dir/abbd398c-ad2f-4d13-91d0-fbcf97802571-dshm\") pod \"abbd398c-ad2f-4d13-91d0-fbcf97802571\" (UID: \"abbd398c-ad2f-4d13-91d0-fbcf97802571\") " Nov 02 18:12:22.436438 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:22.436402 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abbd398c-ad2f-4d13-91d0-fbcf97802571-kube-api-access-qd2tx\" (UniqueName: \"kubernetes.io/projected/abbd398c-ad2f-4d13-91d0-fbcf97802571-kube-api-access-qd2tx\") pod \"abbd398c-ad2f-4d13-91d0-fbcf97802571\" (UID: \"abbd398c-ad2f-4d13-91d0-fbcf97802571\") " Nov 02 18:12:22.436634 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:22.436448 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abbd398c-ad2f-4d13-91d0-fbcf97802571-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abbd398c-ad2f-4d13-91d0-fbcf97802571-model-cache\") pod \"abbd398c-ad2f-4d13-91d0-fbcf97802571\" (UID: \"abbd398c-ad2f-4d13-91d0-fbcf97802571\") " Nov 02 18:12:22.436634 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:22.436475 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abbd398c-ad2f-4d13-91d0-fbcf97802571-home" pod "abbd398c-ad2f-4d13-91d0-fbcf97802571" (UID: "abbd398c-ad2f-4d13-91d0-fbcf97802571"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:22.436634 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:22.436541 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abbd398c-ad2f-4d13-91d0-fbcf97802571-tls-certs\" (UniqueName: \"kubernetes.io/secret/abbd398c-ad2f-4d13-91d0-fbcf97802571-tls-certs\") pod \"abbd398c-ad2f-4d13-91d0-fbcf97802571\" (UID: \"abbd398c-ad2f-4d13-91d0-fbcf97802571\") " Nov 02 18:12:22.436634 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:22.436573 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abbd398c-ad2f-4d13-91d0-fbcf97802571-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abbd398c-ad2f-4d13-91d0-fbcf97802571-tmp-dir\") pod \"abbd398c-ad2f-4d13-91d0-fbcf97802571\" (UID: \"abbd398c-ad2f-4d13-91d0-fbcf97802571\") " Nov 02 18:12:22.436877 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:22.436829 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abbd398c-ad2f-4d13-91d0-fbcf97802571-model-cache" pod "abbd398c-ad2f-4d13-91d0-fbcf97802571" (UID: "abbd398c-ad2f-4d13-91d0-fbcf97802571"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:22.437230 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:22.437207 4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abbd398c-ad2f-4d13-91d0-fbcf97802571-home\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:22.437387 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:22.437231 4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abbd398c-ad2f-4d13-91d0-fbcf97802571-model-cache\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:22.437520 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:22.437495 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abbd398c-ad2f-4d13-91d0-fbcf97802571-tmp-dir" pod "abbd398c-ad2f-4d13-91d0-fbcf97802571" (UID: "abbd398c-ad2f-4d13-91d0-fbcf97802571"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:22.438794 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:22.438765 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abbd398c-ad2f-4d13-91d0-fbcf97802571-dshm" pod "abbd398c-ad2f-4d13-91d0-fbcf97802571" (UID: "abbd398c-ad2f-4d13-91d0-fbcf97802571"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:22.439076 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:22.439055 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abbd398c-ad2f-4d13-91d0-fbcf97802571-tls-certs" pod "abbd398c-ad2f-4d13-91d0-fbcf97802571" (UID: "abbd398c-ad2f-4d13-91d0-fbcf97802571"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:12:22.439130 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:22.439090 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abbd398c-ad2f-4d13-91d0-fbcf97802571-kube-api-access-qd2tx" pod "abbd398c-ad2f-4d13-91d0-fbcf97802571" (UID: "abbd398c-ad2f-4d13-91d0-fbcf97802571"). InnerVolumeSpecName "kube-api-access-qd2tx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:22.538791 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:22.538759 4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abbd398c-ad2f-4d13-91d0-fbcf97802571-tls-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:22.538791 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:22.538789 4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abbd398c-ad2f-4d13-91d0-fbcf97802571-tmp-dir\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:22.538791 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:22.538797 4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abbd398c-ad2f-4d13-91d0-fbcf97802571-dshm\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:22.539038 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:22.538807 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd2tx\" (UniqueName: \"kubernetes.io/projected/abbd398c-ad2f-4d13-91d0-fbcf97802571-kube-api-access-qd2tx\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:22.760666 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:22.760629 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:12:22.798663 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:22.798582 4471 generic.go:356] "Generic (PLEG): container finished" podID="abbd398c-ad2f-4d13-91d0-fbcf97802571" containerID="721d153d11b375908cdfa847071e43d44cb8544303522bc1fca5a8349f46731b" exitCode=0 Nov 02 18:12:22.798663 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:22.798645 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5" Nov 02 18:12:22.798874 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:22.798664 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5" event={"ID":"abbd398c-ad2f-4d13-91d0-fbcf97802571","Type":"ContainerDied","Data":"721d153d11b375908cdfa847071e43d44cb8544303522bc1fca5a8349f46731b"} Nov 02 18:12:22.798874 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:22.798705 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5" event={"ID":"abbd398c-ad2f-4d13-91d0-fbcf97802571","Type":"ContainerDied","Data":"22a4ef907eba6a1343d809b3333958721b1189014dc926e60f0916e28b409a2c"} Nov 02 18:12:22.798874 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:22.798722 4471 scope.go:122] "RemoveContainer" containerID="721d153d11b375908cdfa847071e43d44cb8544303522bc1fca5a8349f46731b" Nov 02 18:12:22.812853 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:22.812826 4471 scope.go:122] "RemoveContainer" containerID="721d153d11b375908cdfa847071e43d44cb8544303522bc1fca5a8349f46731b" Nov 02 18:12:22.813398 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:12:22.813369 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"721d153d11b375908cdfa847071e43d44cb8544303522bc1fca5a8349f46731b\": container with ID starting with 721d153d11b375908cdfa847071e43d44cb8544303522bc1fca5a8349f46731b not found: ID does not exist" containerID="721d153d11b375908cdfa847071e43d44cb8544303522bc1fca5a8349f46731b" Nov 02 18:12:22.813486 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:22.813411 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"721d153d11b375908cdfa847071e43d44cb8544303522bc1fca5a8349f46731b"} err="failed to get container status \"721d153d11b375908cdfa847071e43d44cb8544303522bc1fca5a8349f46731b\": rpc error: code = NotFound desc = could not find container \"721d153d11b375908cdfa847071e43d44cb8544303522bc1fca5a8349f46731b\": container with ID starting with 721d153d11b375908cdfa847071e43d44cb8544303522bc1fca5a8349f46731b not found: ID does not exist" Nov 02 18:12:22.856072 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:22.856044 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5"] Nov 02 18:12:22.860330 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:22.860292 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-557feb/auth-test-model-557feb-kserve-55948fb75c-cxhd5"] Nov 02 18:12:24.139277 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:24.139238 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg"] Nov 02 18:12:24.139759 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:24.139541 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg" podUID="db82dff8-26f4-43f9-a544-aa605d6f5ac4" containerName="maas-api" containerID="cri-o://e1fd9a0f9f3a3bbaaa2c67ac9149b0569efd251315c6597f96f43fd29ffdcea0" gracePeriod=30 Nov 02 18:12:24.461388 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:24.461240 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abbd398c-ad2f-4d13-91d0-fbcf97802571" path="/var/lib/kubelet/pods/abbd398c-ad2f-4d13-91d0-fbcf97802571/volumes" Nov 02 18:12:24.506329 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:24.506215 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg" Nov 02 18:12:24.673410 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:24.672809 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db82dff8-26f4-43f9-a544-aa605d6f5ac4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db82dff8-26f4-43f9-a544-aa605d6f5ac4-openshift-service-ca\") pod \"db82dff8-26f4-43f9-a544-aa605d6f5ac4\" (UID: \"db82dff8-26f4-43f9-a544-aa605d6f5ac4\") " Nov 02 18:12:24.673410 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:24.672849 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db82dff8-26f4-43f9-a544-aa605d6f5ac4-kube-api-access-ps4n4\" (UniqueName: \"kubernetes.io/projected/db82dff8-26f4-43f9-a544-aa605d6f5ac4-kube-api-access-ps4n4\") pod \"db82dff8-26f4-43f9-a544-aa605d6f5ac4\" (UID: \"db82dff8-26f4-43f9-a544-aa605d6f5ac4\") " Nov 02 18:12:24.673410 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:24.673340 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db82dff8-26f4-43f9-a544-aa605d6f5ac4-openshift-service-ca" pod "db82dff8-26f4-43f9-a544-aa605d6f5ac4" (UID: "db82dff8-26f4-43f9-a544-aa605d6f5ac4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:24.673751 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:24.673675 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db82dff8-26f4-43f9-a544-aa605d6f5ac4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db82dff8-26f4-43f9-a544-aa605d6f5ac4-maas-api-tls\") pod \"db82dff8-26f4-43f9-a544-aa605d6f5ac4\" (UID: \"db82dff8-26f4-43f9-a544-aa605d6f5ac4\") " Nov 02 18:12:24.673823 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:24.673760 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db82dff8-26f4-43f9-a544-aa605d6f5ac4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/db82dff8-26f4-43f9-a544-aa605d6f5ac4-metrics-certs\") pod \"db82dff8-26f4-43f9-a544-aa605d6f5ac4\" (UID: \"db82dff8-26f4-43f9-a544-aa605d6f5ac4\") " Nov 02 18:12:24.679338 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:24.674638 4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db82dff8-26f4-43f9-a544-aa605d6f5ac4-openshift-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:24.679338 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:24.676796 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db82dff8-26f4-43f9-a544-aa605d6f5ac4-kube-api-access-ps4n4" pod "db82dff8-26f4-43f9-a544-aa605d6f5ac4" (UID: "db82dff8-26f4-43f9-a544-aa605d6f5ac4"). InnerVolumeSpecName "kube-api-access-ps4n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:24.679338 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:24.677014 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db82dff8-26f4-43f9-a544-aa605d6f5ac4-maas-api-tls" pod "db82dff8-26f4-43f9-a544-aa605d6f5ac4" (UID: "db82dff8-26f4-43f9-a544-aa605d6f5ac4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:12:24.679338 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:24.678620 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db82dff8-26f4-43f9-a544-aa605d6f5ac4-metrics-certs" pod "db82dff8-26f4-43f9-a544-aa605d6f5ac4" (UID: "db82dff8-26f4-43f9-a544-aa605d6f5ac4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:12:24.776271 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:24.776221 4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db82dff8-26f4-43f9-a544-aa605d6f5ac4-metrics-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:24.776271 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:24.776267 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps4n4\" (UniqueName: \"kubernetes.io/projected/db82dff8-26f4-43f9-a544-aa605d6f5ac4-kube-api-access-ps4n4\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:24.776579 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:24.776282 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db82dff8-26f4-43f9-a544-aa605d6f5ac4-maas-api-tls\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:24.815826 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:24.815738 4471 generic.go:356] "Generic (PLEG): container finished" podID="db82dff8-26f4-43f9-a544-aa605d6f5ac4" containerID="e1fd9a0f9f3a3bbaaa2c67ac9149b0569efd251315c6597f96f43fd29ffdcea0" exitCode=0 Nov 02 18:12:24.815826 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:24.815789 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg" Nov 02 18:12:24.816079 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:24.815823 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg" event={"ID":"db82dff8-26f4-43f9-a544-aa605d6f5ac4","Type":"ContainerDied","Data":"e1fd9a0f9f3a3bbaaa2c67ac9149b0569efd251315c6597f96f43fd29ffdcea0"} Nov 02 18:12:24.816079 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:24.815882 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg" event={"ID":"db82dff8-26f4-43f9-a544-aa605d6f5ac4","Type":"ContainerDied","Data":"9caf0fcf659457dd25e3609f416077a64830ac0296506dc9d10463b114eefb12"} Nov 02 18:12:24.816079 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:24.815912 4471 scope.go:122] "RemoveContainer" containerID="e1fd9a0f9f3a3bbaaa2c67ac9149b0569efd251315c6597f96f43fd29ffdcea0" Nov 02 18:12:24.827489 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:24.827469 4471 scope.go:122] "RemoveContainer" containerID="e1fd9a0f9f3a3bbaaa2c67ac9149b0569efd251315c6597f96f43fd29ffdcea0" Nov 02 18:12:24.827740 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:12:24.827722 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1fd9a0f9f3a3bbaaa2c67ac9149b0569efd251315c6597f96f43fd29ffdcea0\": container with ID starting with e1fd9a0f9f3a3bbaaa2c67ac9149b0569efd251315c6597f96f43fd29ffdcea0 not found: ID does not exist" containerID="e1fd9a0f9f3a3bbaaa2c67ac9149b0569efd251315c6597f96f43fd29ffdcea0" Nov 02 18:12:24.827810 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:24.827747 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1fd9a0f9f3a3bbaaa2c67ac9149b0569efd251315c6597f96f43fd29ffdcea0"} err="failed to get container status \"e1fd9a0f9f3a3bbaaa2c67ac9149b0569efd251315c6597f96f43fd29ffdcea0\": rpc error: code = NotFound desc = could not find container \"e1fd9a0f9f3a3bbaaa2c67ac9149b0569efd251315c6597f96f43fd29ffdcea0\": container with ID starting with e1fd9a0f9f3a3bbaaa2c67ac9149b0569efd251315c6597f96f43fd29ffdcea0 not found: ID does not exist" Nov 02 18:12:24.888551 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:24.888514 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg"] Nov 02 18:12:24.984363 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:24.975374 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b91583-7764dd8bbd-snxmg"] Nov 02 18:12:25.008730 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:25.008679 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8" podUID="14d9d576-5cbc-4938-82ec-34191e9675a5" containerName="main" containerID="cri-o://e0c124a9e77296660ce1306cd1a02806cc7e72aedb00e2ba550d77e72539b529" gracePeriod=45 Nov 02 18:12:25.286416 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:25.286342 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8" Nov 02 18:12:25.387242 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:25.387205 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14d9d576-5cbc-4938-82ec-34191e9675a5-home\" (UniqueName: \"kubernetes.io/empty-dir/14d9d576-5cbc-4938-82ec-34191e9675a5-home\") pod \"14d9d576-5cbc-4938-82ec-34191e9675a5\" (UID: \"14d9d576-5cbc-4938-82ec-34191e9675a5\") " Nov 02 18:12:25.387442 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:25.387261 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14d9d576-5cbc-4938-82ec-34191e9675a5-kube-api-access-9gmbt\" (UniqueName: \"kubernetes.io/projected/14d9d576-5cbc-4938-82ec-34191e9675a5-kube-api-access-9gmbt\") pod \"14d9d576-5cbc-4938-82ec-34191e9675a5\" (UID: \"14d9d576-5cbc-4938-82ec-34191e9675a5\") " Nov 02 18:12:25.387442 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:25.387300 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14d9d576-5cbc-4938-82ec-34191e9675a5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14d9d576-5cbc-4938-82ec-34191e9675a5-model-cache\") pod \"14d9d576-5cbc-4938-82ec-34191e9675a5\" (UID: \"14d9d576-5cbc-4938-82ec-34191e9675a5\") " Nov 02 18:12:25.387442 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:25.387433 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14d9d576-5cbc-4938-82ec-34191e9675a5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/14d9d576-5cbc-4938-82ec-34191e9675a5-dshm\") pod \"14d9d576-5cbc-4938-82ec-34191e9675a5\" (UID: \"14d9d576-5cbc-4938-82ec-34191e9675a5\") " Nov 02 18:12:25.387585 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:25.387469 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14d9d576-5cbc-4938-82ec-34191e9675a5-tls-certs\" (UniqueName: \"kubernetes.io/secret/14d9d576-5cbc-4938-82ec-34191e9675a5-tls-certs\") pod \"14d9d576-5cbc-4938-82ec-34191e9675a5\" (UID: \"14d9d576-5cbc-4938-82ec-34191e9675a5\") " Nov 02 18:12:25.387585 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:25.387530 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14d9d576-5cbc-4938-82ec-34191e9675a5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14d9d576-5cbc-4938-82ec-34191e9675a5-tmp-dir\") pod \"14d9d576-5cbc-4938-82ec-34191e9675a5\" (UID: \"14d9d576-5cbc-4938-82ec-34191e9675a5\") " Nov 02 18:12:25.387585 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:25.387531 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14d9d576-5cbc-4938-82ec-34191e9675a5-home" pod "14d9d576-5cbc-4938-82ec-34191e9675a5" (UID: "14d9d576-5cbc-4938-82ec-34191e9675a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:25.387751 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:25.387710 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14d9d576-5cbc-4938-82ec-34191e9675a5-model-cache" pod "14d9d576-5cbc-4938-82ec-34191e9675a5" (UID: "14d9d576-5cbc-4938-82ec-34191e9675a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:25.387869 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:25.387841 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14d9d576-5cbc-4938-82ec-34191e9675a5-tmp-dir" pod "14d9d576-5cbc-4938-82ec-34191e9675a5" (UID: "14d9d576-5cbc-4938-82ec-34191e9675a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:25.388081 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:25.388065 4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14d9d576-5cbc-4938-82ec-34191e9675a5-tmp-dir\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:25.388155 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:25.388082 4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14d9d576-5cbc-4938-82ec-34191e9675a5-home\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:25.388155 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:25.388095 4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14d9d576-5cbc-4938-82ec-34191e9675a5-model-cache\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:25.389742 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:25.389721 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d9d576-5cbc-4938-82ec-34191e9675a5-kube-api-access-9gmbt" pod "14d9d576-5cbc-4938-82ec-34191e9675a5" (UID: "14d9d576-5cbc-4938-82ec-34191e9675a5"). InnerVolumeSpecName "kube-api-access-9gmbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:25.389830 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:25.389741 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14d9d576-5cbc-4938-82ec-34191e9675a5-dshm" pod "14d9d576-5cbc-4938-82ec-34191e9675a5" (UID: "14d9d576-5cbc-4938-82ec-34191e9675a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:25.389830 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:25.389750 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14d9d576-5cbc-4938-82ec-34191e9675a5-tls-certs" pod "14d9d576-5cbc-4938-82ec-34191e9675a5" (UID: "14d9d576-5cbc-4938-82ec-34191e9675a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:12:25.489186 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:25.489151 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gmbt\" (UniqueName: \"kubernetes.io/projected/14d9d576-5cbc-4938-82ec-34191e9675a5-kube-api-access-9gmbt\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:25.489186 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:25.489184 4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14d9d576-5cbc-4938-82ec-34191e9675a5-dshm\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:25.489186 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:25.489195 4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14d9d576-5cbc-4938-82ec-34191e9675a5-tls-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:25.825261 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:25.825223 4471 generic.go:356] "Generic (PLEG): container finished" podID="14d9d576-5cbc-4938-82ec-34191e9675a5" containerID="e0c124a9e77296660ce1306cd1a02806cc7e72aedb00e2ba550d77e72539b529" exitCode=0 Nov 02 18:12:25.825511 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:25.825282 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8" Nov 02 18:12:25.825511 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:25.825322 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8" event={"ID":"14d9d576-5cbc-4938-82ec-34191e9675a5","Type":"ContainerDied","Data":"e0c124a9e77296660ce1306cd1a02806cc7e72aedb00e2ba550d77e72539b529"} Nov 02 18:12:25.825511 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:25.825371 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8" event={"ID":"14d9d576-5cbc-4938-82ec-34191e9675a5","Type":"ContainerDied","Data":"430f944cc9079f9fb50cb014618c9e86e3827a1de5e6abb8c2f9369613b70c59"} Nov 02 18:12:25.825511 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:25.825389 4471 scope.go:122] "RemoveContainer" containerID="e0c124a9e77296660ce1306cd1a02806cc7e72aedb00e2ba550d77e72539b529" Nov 02 18:12:25.836185 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:25.836168 4471 scope.go:122] "RemoveContainer" containerID="e0c124a9e77296660ce1306cd1a02806cc7e72aedb00e2ba550d77e72539b529" Nov 02 18:12:25.836475 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:12:25.836456 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0c124a9e77296660ce1306cd1a02806cc7e72aedb00e2ba550d77e72539b529\": container with ID starting with e0c124a9e77296660ce1306cd1a02806cc7e72aedb00e2ba550d77e72539b529 not found: ID does not exist" containerID="e0c124a9e77296660ce1306cd1a02806cc7e72aedb00e2ba550d77e72539b529" Nov 02 18:12:25.836529 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:25.836484 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0c124a9e77296660ce1306cd1a02806cc7e72aedb00e2ba550d77e72539b529"} err="failed to get container status \"e0c124a9e77296660ce1306cd1a02806cc7e72aedb00e2ba550d77e72539b529\": rpc error: code = NotFound desc = could not find container \"e0c124a9e77296660ce1306cd1a02806cc7e72aedb00e2ba550d77e72539b529\": container with ID starting with e0c124a9e77296660ce1306cd1a02806cc7e72aedb00e2ba550d77e72539b529 not found: ID does not exist" Nov 02 18:12:25.885874 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:25.885841 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8"] Nov 02 18:12:25.898361 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:25.898322 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-227939/auth-test-model-227939-kserve-69d8bb7cc-4fsx8"] Nov 02 18:12:26.441224 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:26.441193 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14d9d576-5cbc-4938-82ec-34191e9675a5" path="/var/lib/kubelet/pods/14d9d576-5cbc-4938-82ec-34191e9675a5/volumes" Nov 02 18:12:26.441724 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:26.441707 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db82dff8-26f4-43f9-a544-aa605d6f5ac4" path="/var/lib/kubelet/pods/db82dff8-26f4-43f9-a544-aa605d6f5ac4/volumes" Nov 02 18:12:28.893148 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:28.893114 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp"] Nov 02 18:12:28.893616 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:28.893390 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp" podUID="8627beff-73fc-4a24-a846-55d6a27ce9b7" containerName="maas-api" containerID="cri-o://af1cb59d1989dd264675a54dede81d2f12bfefe92a4a5aa15daee50960577b62" gracePeriod=30 Nov 02 18:12:29.258990 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:29.258965 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp" Nov 02 18:12:29.286123 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:29.286088 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8627beff-73fc-4a24-a846-55d6a27ce9b7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8627beff-73fc-4a24-a846-55d6a27ce9b7-openshift-service-ca\") pod \"8627beff-73fc-4a24-a846-55d6a27ce9b7\" (UID: \"8627beff-73fc-4a24-a846-55d6a27ce9b7\") " Nov 02 18:12:29.286428 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:29.286399 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8627beff-73fc-4a24-a846-55d6a27ce9b7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8627beff-73fc-4a24-a846-55d6a27ce9b7-maas-api-tls\") pod \"8627beff-73fc-4a24-a846-55d6a27ce9b7\" (UID: \"8627beff-73fc-4a24-a846-55d6a27ce9b7\") " Nov 02 18:12:29.286563 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:29.286504 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8627beff-73fc-4a24-a846-55d6a27ce9b7-kube-api-access-4qd8d\" (UniqueName: \"kubernetes.io/projected/8627beff-73fc-4a24-a846-55d6a27ce9b7-kube-api-access-4qd8d\") pod \"8627beff-73fc-4a24-a846-55d6a27ce9b7\" (UID: \"8627beff-73fc-4a24-a846-55d6a27ce9b7\") " Nov 02 18:12:29.286745 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:29.286718 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8627beff-73fc-4a24-a846-55d6a27ce9b7-openshift-service-ca" pod "8627beff-73fc-4a24-a846-55d6a27ce9b7" (UID: "8627beff-73fc-4a24-a846-55d6a27ce9b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:29.287098 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:29.287051 4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8627beff-73fc-4a24-a846-55d6a27ce9b7-openshift-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:29.289231 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:29.289208 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8627beff-73fc-4a24-a846-55d6a27ce9b7-kube-api-access-4qd8d" pod "8627beff-73fc-4a24-a846-55d6a27ce9b7" (UID: "8627beff-73fc-4a24-a846-55d6a27ce9b7"). InnerVolumeSpecName "kube-api-access-4qd8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:29.289453 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:29.289434 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8627beff-73fc-4a24-a846-55d6a27ce9b7-maas-api-tls" pod "8627beff-73fc-4a24-a846-55d6a27ce9b7" (UID: "8627beff-73fc-4a24-a846-55d6a27ce9b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:12:29.390826 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:29.389904 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8627beff-73fc-4a24-a846-55d6a27ce9b7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8627beff-73fc-4a24-a846-55d6a27ce9b7-metrics-certs\") pod \"8627beff-73fc-4a24-a846-55d6a27ce9b7\" (UID: \"8627beff-73fc-4a24-a846-55d6a27ce9b7\") " Nov 02 18:12:29.390826 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:29.390430 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8627beff-73fc-4a24-a846-55d6a27ce9b7-maas-api-tls\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:29.390826 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:29.390451 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qd8d\" (UniqueName: \"kubernetes.io/projected/8627beff-73fc-4a24-a846-55d6a27ce9b7-kube-api-access-4qd8d\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:29.393561 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:29.393527 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8627beff-73fc-4a24-a846-55d6a27ce9b7-metrics-certs" pod "8627beff-73fc-4a24-a846-55d6a27ce9b7" (UID: "8627beff-73fc-4a24-a846-55d6a27ce9b7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:12:29.491777 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:29.491693 4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8627beff-73fc-4a24-a846-55d6a27ce9b7-metrics-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:29.861471 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:29.858088 4471 generic.go:356] "Generic (PLEG): container finished" podID="8627beff-73fc-4a24-a846-55d6a27ce9b7" containerID="af1cb59d1989dd264675a54dede81d2f12bfefe92a4a5aa15daee50960577b62" exitCode=0 Nov 02 18:12:29.861471 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:29.858183 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp" event={"ID":"8627beff-73fc-4a24-a846-55d6a27ce9b7","Type":"ContainerDied","Data":"af1cb59d1989dd264675a54dede81d2f12bfefe92a4a5aa15daee50960577b62"} Nov 02 18:12:29.861471 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:29.858230 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp" event={"ID":"8627beff-73fc-4a24-a846-55d6a27ce9b7","Type":"ContainerDied","Data":"205c4967cb3ef0c96d330857e99cb2e782a8a79b2434443a7f284ab2ea3d5dfe"} Nov 02 18:12:29.861471 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:29.858256 4471 scope.go:122] "RemoveContainer" containerID="af1cb59d1989dd264675a54dede81d2f12bfefe92a4a5aa15daee50960577b62" Nov 02 18:12:29.861471 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:29.858391 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp" Nov 02 18:12:29.886757 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:29.886715 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-120394-854c949567-5xfqn"] Nov 02 18:12:29.886989 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:29.886960 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-120394-854c949567-5xfqn" podUID="7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9" containerName="maas-api" containerID="cri-o://e584069d5f2542331f41acfb8ff40e1669c901fb178f7d100ce666f2868873ff" gracePeriod=30 Nov 02 18:12:29.902743 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:29.902471 4471 scope.go:122] "RemoveContainer" containerID="af1cb59d1989dd264675a54dede81d2f12bfefe92a4a5aa15daee50960577b62" Nov 02 18:12:29.903228 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:12:29.903194 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af1cb59d1989dd264675a54dede81d2f12bfefe92a4a5aa15daee50960577b62\": container with ID starting with af1cb59d1989dd264675a54dede81d2f12bfefe92a4a5aa15daee50960577b62 not found: ID does not exist" containerID="af1cb59d1989dd264675a54dede81d2f12bfefe92a4a5aa15daee50960577b62" Nov 02 18:12:29.903269 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:29.903242 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af1cb59d1989dd264675a54dede81d2f12bfefe92a4a5aa15daee50960577b62"} err="failed to get container status \"af1cb59d1989dd264675a54dede81d2f12bfefe92a4a5aa15daee50960577b62\": rpc error: code = NotFound desc = could not find container \"af1cb59d1989dd264675a54dede81d2f12bfefe92a4a5aa15daee50960577b62\": container with ID starting with af1cb59d1989dd264675a54dede81d2f12bfefe92a4a5aa15daee50960577b62 not found: ID does not exist" Nov 02 18:12:29.919131 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:29.919103 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp"] Nov 02 18:12:29.926283 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:29.926259 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a29999-bcbf7cc9f-jgxzp"] Nov 02 18:12:30.241723 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:30.241691 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-120394-854c949567-5xfqn" Nov 02 18:12:30.312995 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:30.312960 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9-kube-api-access-pzk44\" (UniqueName: \"kubernetes.io/projected/7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9-kube-api-access-pzk44\") pod \"7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9\" (UID: \"7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9\") " Nov 02 18:12:30.313198 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:30.313027 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9-metrics-certs\") pod \"7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9\" (UID: \"7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9\") " Nov 02 18:12:30.313198 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:30.313110 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9-maas-api-tls\") pod \"7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9\" (UID: \"7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9\") " Nov 02 18:12:30.313198 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:30.313149 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9-openshift-service-ca\") pod \"7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9\" (UID: \"7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9\") " Nov 02 18:12:30.313621 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:30.313594 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9-openshift-service-ca" pod "7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9" (UID: "7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:30.315263 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:30.315231 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9-maas-api-tls" pod "7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9" (UID: "7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:12:30.315438 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:30.315292 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9-kube-api-access-pzk44" pod "7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9" (UID: "7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9"). InnerVolumeSpecName "kube-api-access-pzk44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:30.315438 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:30.315365 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9-metrics-certs" pod "7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9" (UID: "7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:12:30.470437 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:30.468770 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzk44\" (UniqueName: \"kubernetes.io/projected/7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9-kube-api-access-pzk44\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:30.470437 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:30.468809 4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9-metrics-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:30.470437 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:30.468825 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9-maas-api-tls\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:30.470437 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:30.468840 4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9-openshift-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:30.549390 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:30.548347 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8627beff-73fc-4a24-a846-55d6a27ce9b7" path="/var/lib/kubelet/pods/8627beff-73fc-4a24-a846-55d6a27ce9b7/volumes" Nov 02 18:12:30.867777 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:30.867740 4471 generic.go:356] "Generic (PLEG): container finished" podID="7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9" containerID="e584069d5f2542331f41acfb8ff40e1669c901fb178f7d100ce666f2868873ff" exitCode=0 Nov 02 18:12:30.867958 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:30.867797 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-120394-854c949567-5xfqn" Nov 02 18:12:30.867958 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:30.867801 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-120394-854c949567-5xfqn" event={"ID":"7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9","Type":"ContainerDied","Data":"e584069d5f2542331f41acfb8ff40e1669c901fb178f7d100ce666f2868873ff"} Nov 02 18:12:30.867958 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:30.867939 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-120394-854c949567-5xfqn" event={"ID":"7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9","Type":"ContainerDied","Data":"c0902f4d39ce1089949658a8c2d37e78400e2c711f84e04309d0d7ce0f696d8c"} Nov 02 18:12:30.868071 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:30.867977 4471 scope.go:122] "RemoveContainer" containerID="e584069d5f2542331f41acfb8ff40e1669c901fb178f7d100ce666f2868873ff" Nov 02 18:12:30.878122 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:30.878095 4471 scope.go:122] "RemoveContainer" containerID="e584069d5f2542331f41acfb8ff40e1669c901fb178f7d100ce666f2868873ff" Nov 02 18:12:30.878418 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:12:30.878394 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e584069d5f2542331f41acfb8ff40e1669c901fb178f7d100ce666f2868873ff\": container with ID starting with e584069d5f2542331f41acfb8ff40e1669c901fb178f7d100ce666f2868873ff not found: ID does not exist" containerID="e584069d5f2542331f41acfb8ff40e1669c901fb178f7d100ce666f2868873ff" Nov 02 18:12:30.878484 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:30.878427 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e584069d5f2542331f41acfb8ff40e1669c901fb178f7d100ce666f2868873ff"} err="failed to get container status \"e584069d5f2542331f41acfb8ff40e1669c901fb178f7d100ce666f2868873ff\": rpc error: code = NotFound desc = could not find container \"e584069d5f2542331f41acfb8ff40e1669c901fb178f7d100ce666f2868873ff\": container with ID starting with e584069d5f2542331f41acfb8ff40e1669c901fb178f7d100ce666f2868873ff not found: ID does not exist" Nov 02 18:12:30.922118 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:30.922087 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-120394-854c949567-5xfqn"] Nov 02 18:12:30.927863 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:30.927835 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-120394-854c949567-5xfqn"] Nov 02 18:12:31.439323 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:31.428923 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq"] Nov 02 18:12:31.439323 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:31.429187 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq" podUID="54ac053b-bdfa-4471-8799-aabab39ebe94" containerName="maas-api" containerID="cri-o://4a86a253920ec27bf194236708fe19a0fe8421ab2dc30c34b1727e16c8ca6f39" gracePeriod=30 Nov 02 18:12:31.735161 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:31.735135 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq" Nov 02 18:12:31.841946 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:31.836839 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54ac053b-bdfa-4471-8799-aabab39ebe94-metrics-certs\" (UniqueName: \"kubernetes.io/secret/54ac053b-bdfa-4471-8799-aabab39ebe94-metrics-certs\") pod \"54ac053b-bdfa-4471-8799-aabab39ebe94\" (UID: \"54ac053b-bdfa-4471-8799-aabab39ebe94\") " Nov 02 18:12:31.841946 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:31.836909 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54ac053b-bdfa-4471-8799-aabab39ebe94-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54ac053b-bdfa-4471-8799-aabab39ebe94-openshift-service-ca\") pod \"54ac053b-bdfa-4471-8799-aabab39ebe94\" (UID: \"54ac053b-bdfa-4471-8799-aabab39ebe94\") " Nov 02 18:12:31.841946 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:31.837053 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54ac053b-bdfa-4471-8799-aabab39ebe94-kube-api-access-2npjm\" (UniqueName: \"kubernetes.io/projected/54ac053b-bdfa-4471-8799-aabab39ebe94-kube-api-access-2npjm\") pod \"54ac053b-bdfa-4471-8799-aabab39ebe94\" (UID: \"54ac053b-bdfa-4471-8799-aabab39ebe94\") " Nov 02 18:12:31.841946 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:31.837085 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54ac053b-bdfa-4471-8799-aabab39ebe94-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54ac053b-bdfa-4471-8799-aabab39ebe94-maas-api-tls\") pod \"54ac053b-bdfa-4471-8799-aabab39ebe94\" (UID: \"54ac053b-bdfa-4471-8799-aabab39ebe94\") " Nov 02 18:12:31.841946 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:31.839004 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ac053b-bdfa-4471-8799-aabab39ebe94-openshift-service-ca" pod "54ac053b-bdfa-4471-8799-aabab39ebe94" (UID: "54ac053b-bdfa-4471-8799-aabab39ebe94"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:31.842747 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:31.842657 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ac053b-bdfa-4471-8799-aabab39ebe94-maas-api-tls" pod "54ac053b-bdfa-4471-8799-aabab39ebe94" (UID: "54ac053b-bdfa-4471-8799-aabab39ebe94"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:12:31.847666 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:31.847626 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ac053b-bdfa-4471-8799-aabab39ebe94-metrics-certs" pod "54ac053b-bdfa-4471-8799-aabab39ebe94" (UID: "54ac053b-bdfa-4471-8799-aabab39ebe94"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:12:31.888327 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:31.879185 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ac053b-bdfa-4471-8799-aabab39ebe94-kube-api-access-2npjm" pod "54ac053b-bdfa-4471-8799-aabab39ebe94" (UID: "54ac053b-bdfa-4471-8799-aabab39ebe94"). InnerVolumeSpecName "kube-api-access-2npjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:31.908732 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:31.907754 4471 generic.go:356] "Generic (PLEG): container finished" podID="54ac053b-bdfa-4471-8799-aabab39ebe94" containerID="4a86a253920ec27bf194236708fe19a0fe8421ab2dc30c34b1727e16c8ca6f39" exitCode=0 Nov 02 18:12:31.908732 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:31.907890 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq" event={"ID":"54ac053b-bdfa-4471-8799-aabab39ebe94","Type":"ContainerDied","Data":"4a86a253920ec27bf194236708fe19a0fe8421ab2dc30c34b1727e16c8ca6f39"} Nov 02 18:12:31.908732 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:31.907936 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq" event={"ID":"54ac053b-bdfa-4471-8799-aabab39ebe94","Type":"ContainerDied","Data":"b06ee8c131663c2923bde6633a5e8ccafd18dd084dce9bbc2b2e0168be350945"} Nov 02 18:12:31.908732 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:31.907959 4471 scope.go:122] "RemoveContainer" containerID="4a86a253920ec27bf194236708fe19a0fe8421ab2dc30c34b1727e16c8ca6f39" Nov 02 18:12:31.908732 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:31.908058 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq" Nov 02 18:12:31.920806 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:31.920783 4471 scope.go:122] "RemoveContainer" containerID="4a86a253920ec27bf194236708fe19a0fe8421ab2dc30c34b1727e16c8ca6f39" Nov 02 18:12:31.921088 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:12:31.921060 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a86a253920ec27bf194236708fe19a0fe8421ab2dc30c34b1727e16c8ca6f39\": container with ID starting with 4a86a253920ec27bf194236708fe19a0fe8421ab2dc30c34b1727e16c8ca6f39 not found: ID does not exist" containerID="4a86a253920ec27bf194236708fe19a0fe8421ab2dc30c34b1727e16c8ca6f39" Nov 02 18:12:31.921170 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:31.921099 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a86a253920ec27bf194236708fe19a0fe8421ab2dc30c34b1727e16c8ca6f39"} err="failed to get container status \"4a86a253920ec27bf194236708fe19a0fe8421ab2dc30c34b1727e16c8ca6f39\": rpc error: code = NotFound desc = could not find container \"4a86a253920ec27bf194236708fe19a0fe8421ab2dc30c34b1727e16c8ca6f39\": container with ID starting with 4a86a253920ec27bf194236708fe19a0fe8421ab2dc30c34b1727e16c8ca6f39 not found: ID does not exist" Nov 02 18:12:31.939162 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:31.939136 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2npjm\" (UniqueName: \"kubernetes.io/projected/54ac053b-bdfa-4471-8799-aabab39ebe94-kube-api-access-2npjm\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:31.939518 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:31.939163 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54ac053b-bdfa-4471-8799-aabab39ebe94-maas-api-tls\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:31.939518 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:31.939178 4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54ac053b-bdfa-4471-8799-aabab39ebe94-metrics-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:31.939518 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:31.939189 4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54ac053b-bdfa-4471-8799-aabab39ebe94-openshift-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:31.967155 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:31.967094 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq"] Nov 02 18:12:31.969862 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:31.969839 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-557feb-679b865b79-r2ghq"] Nov 02 18:12:32.013607 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.013573 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq"] Nov 02 18:12:32.014190 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014171 4471 cpu_manager.go:414] "containerMap: removing container" podUID="2c047abd-2c48-4b50-ba99-579d0f2cc3fd" containerName="istio-proxy" Nov 02 18:12:32.014190 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014190 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c047abd-2c48-4b50-ba99-579d0f2cc3fd" containerName="istio-proxy" Nov 02 18:12:32.014356 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014204 4471 cpu_manager.go:414] "containerMap: removing container" podUID="ef139666-59eb-4847-9f3d-8fc0d20529dc" containerName="istio-proxy" Nov 02 18:12:32.014356 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014210 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef139666-59eb-4847-9f3d-8fc0d20529dc" containerName="istio-proxy" Nov 02 18:12:32.014356 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014218 4471 cpu_manager.go:414] "containerMap: removing container" podUID="83085553-108f-4d34-b8de-9d0c6e0b13e5" containerName="console" Nov 02 18:12:32.014356 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014223 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83085553-108f-4d34-b8de-9d0c6e0b13e5" containerName="console" Nov 02 18:12:32.014356 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014230 4471 cpu_manager.go:414] "containerMap: removing container" podUID="d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5" containerName="istio-proxy" Nov 02 18:12:32.014356 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014236 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5" containerName="istio-proxy" Nov 02 18:12:32.014356 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014245 4471 cpu_manager.go:414] "containerMap: removing container" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" Nov 02 18:12:32.014356 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014251 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" Nov 02 18:12:32.014356 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014260 4471 cpu_manager.go:414] "containerMap: removing container" podUID="1aa74f62-fe2f-4238-9f58-4a0a20f7892a" containerName="istio-proxy" Nov 02 18:12:32.014356 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014265 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aa74f62-fe2f-4238-9f58-4a0a20f7892a" containerName="istio-proxy" Nov 02 18:12:32.014356 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014274 4471 cpu_manager.go:414] "containerMap: removing container" podUID="abbd398c-ad2f-4d13-91d0-fbcf97802571" containerName="main" Nov 02 18:12:32.014356 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014278 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abbd398c-ad2f-4d13-91d0-fbcf97802571" containerName="main" Nov 02 18:12:32.014356 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014286 4471 cpu_manager.go:414] "containerMap: removing container" podUID="9ebaee6a-d3f4-4f0d-8485-6dd565c1e078" containerName="istio-proxy" Nov 02 18:12:32.014356 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014291 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ebaee6a-d3f4-4f0d-8485-6dd565c1e078" containerName="istio-proxy" Nov 02 18:12:32.014356 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014296 4471 cpu_manager.go:414] "containerMap: removing container" podUID="db82dff8-26f4-43f9-a544-aa605d6f5ac4" containerName="maas-api" Nov 02 18:12:32.014356 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014317 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db82dff8-26f4-43f9-a544-aa605d6f5ac4" containerName="maas-api" Nov 02 18:12:32.014356 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014329 4471 cpu_manager.go:414] "containerMap: removing container" podUID="dd2eb7cb-d5ff-4351-84ac-f19fabf7e616" containerName="payload-pre-processing" Nov 02 18:12:32.014356 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014336 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd2eb7cb-d5ff-4351-84ac-f19fabf7e616" containerName="payload-pre-processing" Nov 02 18:12:32.014356 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014347 4471 cpu_manager.go:414] "containerMap: removing container" podUID="74ec749a-738f-4509-a7e5-be6eb97f4777" containerName="istio-proxy" Nov 02 18:12:32.014356 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014354 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74ec749a-738f-4509-a7e5-be6eb97f4777" containerName="istio-proxy" Nov 02 18:12:32.014356 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014367 4471 cpu_manager.go:414] "containerMap: removing container" podUID="5a819a2f-01ae-45ba-8c60-f3382617c58c" containerName="istio-proxy" Nov 02 18:12:32.014356 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014372 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a819a2f-01ae-45ba-8c60-f3382617c58c" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014378 4471 cpu_manager.go:414] "containerMap: removing container" podUID="d94a6978-2092-4c1d-8731-f18298e5369b" containerName="payload-pre-processing" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014382 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d94a6978-2092-4c1d-8731-f18298e5369b" containerName="payload-pre-processing" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014392 4471 cpu_manager.go:414] "containerMap: removing container" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014396 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014401 4471 cpu_manager.go:414] "containerMap: removing container" podUID="38fea887-fcc0-42fa-a1c9-320e44f70562" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014405 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38fea887-fcc0-42fa-a1c9-320e44f70562" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014413 4471 cpu_manager.go:414] "containerMap: removing container" podUID="341883ea-c8c4-49dd-ba87-ef0a3a539065" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014417 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="341883ea-c8c4-49dd-ba87-ef0a3a539065" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014424 4471 cpu_manager.go:414] "containerMap: removing container" podUID="14d9d576-5cbc-4938-82ec-34191e9675a5" containerName="main" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014428 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14d9d576-5cbc-4938-82ec-34191e9675a5" containerName="main" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014435 4471 cpu_manager.go:414] "containerMap: removing container" podUID="cd908c8a-4026-4da6-95c5-34112ecc019a" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014440 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd908c8a-4026-4da6-95c5-34112ecc019a" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014446 4471 cpu_manager.go:414] "containerMap: removing container" podUID="a06f649d-baff-4ee4-b35b-bc5966df3067" containerName="payload-processing" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014450 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a06f649d-baff-4ee4-b35b-bc5966df3067" containerName="payload-processing" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014457 4471 cpu_manager.go:414] "containerMap: removing container" podUID="e881e4aa-a5ad-46d2-952d-97c5118e98b3" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014461 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e881e4aa-a5ad-46d2-952d-97c5118e98b3" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014469 4471 cpu_manager.go:414] "containerMap: removing container" podUID="54ac053b-bdfa-4471-8799-aabab39ebe94" containerName="maas-api" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014473 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54ac053b-bdfa-4471-8799-aabab39ebe94" containerName="maas-api" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014480 4471 cpu_manager.go:414] "containerMap: removing container" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014484 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014494 4471 cpu_manager.go:414] "containerMap: removing container" podUID="1ac349d9-ff43-493a-8cd2-3bfa0836a8da" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014499 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ac349d9-ff43-493a-8cd2-3bfa0836a8da" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014507 4471 cpu_manager.go:414] "containerMap: removing container" podUID="8627beff-73fc-4a24-a846-55d6a27ce9b7" containerName="maas-api" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014511 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8627beff-73fc-4a24-a846-55d6a27ce9b7" containerName="maas-api" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014526 4471 cpu_manager.go:414] "containerMap: removing container" podUID="6ff6c042-8820-4706-bef8-e4cf2732a389" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014530 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ff6c042-8820-4706-bef8-e4cf2732a389" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014539 4471 cpu_manager.go:414] "containerMap: removing container" podUID="e8eeabf7-f670-46c2-bb59-e07e1a1f1337" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014544 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8eeabf7-f670-46c2-bb59-e07e1a1f1337" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014550 4471 cpu_manager.go:414] "containerMap: removing container" podUID="8210f11c-5046-4e0b-a3ca-d859918e76c7" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014555 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8210f11c-5046-4e0b-a3ca-d859918e76c7" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014562 4471 cpu_manager.go:414] "containerMap: removing container" podUID="7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9" containerName="maas-api" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014566 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9" containerName="maas-api" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014574 4471 cpu_manager.go:414] "containerMap: removing container" podUID="24f05804-b83e-44f0-b01d-b1f34904fba2" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014579 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24f05804-b83e-44f0-b01d-b1f34904fba2" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014584 4471 cpu_manager.go:414] "containerMap: removing container" podUID="5a819a2f-01ae-45ba-8c60-f3382617c58c" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014589 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a819a2f-01ae-45ba-8c60-f3382617c58c" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014678 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ff6c042-8820-4706-bef8-e4cf2732a389" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014688 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8eeabf7-f670-46c2-bb59-e07e1a1f1337" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014694 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="8210f11c-5046-4e0b-a3ca-d859918e76c7" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014700 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9" containerName="maas-api" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014706 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="24f05804-b83e-44f0-b01d-b1f34904fba2" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014711 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a819a2f-01ae-45ba-8c60-f3382617c58c" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014717 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c047abd-2c48-4b50-ba99-579d0f2cc3fd" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014723 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef139666-59eb-4847-9f3d-8fc0d20529dc" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014729 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="83085553-108f-4d34-b8de-9d0c6e0b13e5" containerName="console" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014735 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="d14d1a7f-1dc5-4a6d-893b-03b53bc1cce5" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014741 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ce077ac-aaca-4afc-8090-2fb935b4a3e9" containerName="main" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014747 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aa74f62-fe2f-4238-9f58-4a0a20f7892a" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014752 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="abbd398c-ad2f-4d13-91d0-fbcf97802571" containerName="main" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014757 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ebaee6a-d3f4-4f0d-8485-6dd565c1e078" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014763 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="db82dff8-26f4-43f9-a544-aa605d6f5ac4" containerName="maas-api" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014770 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd2eb7cb-d5ff-4351-84ac-f19fabf7e616" containerName="payload-pre-processing" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014779 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="74ec749a-738f-4509-a7e5-be6eb97f4777" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014788 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="d94a6978-2092-4c1d-8731-f18298e5369b" containerName="payload-pre-processing" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014796 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="520fe9ff-603f-4df8-8b71-06ccfdd1019e" containerName="main" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014801 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="38fea887-fcc0-42fa-a1c9-320e44f70562" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014806 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="341883ea-c8c4-49dd-ba87-ef0a3a539065" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014812 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="14d9d576-5cbc-4938-82ec-34191e9675a5" containerName="main" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014817 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd908c8a-4026-4da6-95c5-34112ecc019a" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014823 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="a06f649d-baff-4ee4-b35b-bc5966df3067" containerName="payload-processing" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014828 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="e881e4aa-a5ad-46d2-952d-97c5118e98b3" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014833 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="54ac053b-bdfa-4471-8799-aabab39ebe94" containerName="maas-api" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014838 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="abaa1ee3-4cc9-47d4-9c44-0ce826b1e496" containerName="main" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014843 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ac349d9-ff43-493a-8cd2-3bfa0836a8da" containerName="istio-proxy" Nov 02 18:12:32.015469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.014849 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="8627beff-73fc-4a24-a846-55d6a27ce9b7" containerName="maas-api" Nov 02 18:12:32.018421 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.018405 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:32.046535 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.046500 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq"] Nov 02 18:12:32.075162 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.074908 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-120394-openshift-default-dockercfg-qq496\"" Nov 02 18:12:32.141243 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.141203 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-workload-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-rctxq\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:32.141420 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.141271 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg9d9\" (UniqueName: \"kubernetes.io/projected/29bc04a6-3a7f-4574-9017-ead0d778ac34-kube-api-access-kg9d9\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-rctxq\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:32.141467 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.141421 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-istio-envoy\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-rctxq\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:32.141512 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.141488 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-workload-certs\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-rctxq\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:32.141564 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.141548 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29bc04a6-3a7f-4574-9017-ead0d778ac34-istiod-ca-cert\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-rctxq\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:32.141602 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.141582 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-credential-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-rctxq\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:32.141664 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.141651 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29bc04a6-3a7f-4574-9017-ead0d778ac34-istio-podinfo\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-rctxq\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:32.141699 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.141673 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29bc04a6-3a7f-4574-9017-ead0d778ac34-istio-token\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-rctxq\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:32.141748 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.141734 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-istio-data\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-rctxq\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:32.243472 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.243380 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-workload-certs\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-rctxq\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:32.243472 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.243456 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29bc04a6-3a7f-4574-9017-ead0d778ac34-istiod-ca-cert\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-rctxq\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:32.243721 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.243486 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-credential-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-rctxq\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:32.243721 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.243529 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29bc04a6-3a7f-4574-9017-ead0d778ac34-istio-podinfo\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-rctxq\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:32.243721 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.243550 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29bc04a6-3a7f-4574-9017-ead0d778ac34-istio-token\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-rctxq\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:32.243721 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.243581 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-istio-data\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-rctxq\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:32.243721 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.243636 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-workload-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-rctxq\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:32.243721 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.243703 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg9d9\" (UniqueName: \"kubernetes.io/projected/29bc04a6-3a7f-4574-9017-ead0d778ac34-kube-api-access-kg9d9\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-rctxq\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:32.244069 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.243784 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-istio-envoy\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-rctxq\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:32.244069 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.243951 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-istio-data\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-rctxq\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:32.244069 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.244057 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-workload-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-rctxq\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:32.244238 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.244102 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-workload-certs\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-rctxq\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:32.244238 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.244206 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-credential-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-rctxq\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:32.244374 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.244321 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29bc04a6-3a7f-4574-9017-ead0d778ac34-istiod-ca-cert\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-rctxq\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:32.246623 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.246590 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-istio-envoy\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-rctxq\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:32.246741 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.246637 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29bc04a6-3a7f-4574-9017-ead0d778ac34-istio-podinfo\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-rctxq\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:32.270129 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.270101 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29bc04a6-3a7f-4574-9017-ead0d778ac34-istio-token\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-rctxq\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:32.270290 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.270273 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg9d9\" (UniqueName: \"kubernetes.io/projected/29bc04a6-3a7f-4574-9017-ead0d778ac34-kube-api-access-kg9d9\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-rctxq\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:32.335330 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.333135 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:32.445229 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.445197 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54ac053b-bdfa-4471-8799-aabab39ebe94" path="/var/lib/kubelet/pods/54ac053b-bdfa-4471-8799-aabab39ebe94/volumes" Nov 02 18:12:32.445854 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.445828 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9" path="/var/lib/kubelet/pods/7d7f0b28-b3a9-4943-b2ae-fa47d66cb4b9/volumes" Nov 02 18:12:32.700628 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.700595 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq"] Nov 02 18:12:32.700835 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:12:32.700813 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29bc04a6_3a7f_4574_9017_ead0d778ac34.slice/crio-99421d3b665e0b338797f494bcc057603b825005e726de8c5b84d91147d37ccc WatchSource:0}: Error finding container 99421d3b665e0b338797f494bcc057603b825005e726de8c5b84d91147d37ccc: Status 404 returned error can't find the container with id 99421d3b665e0b338797f494bcc057603b825005e726de8c5b84d91147d37ccc Nov 02 18:12:32.703814 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.703782 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:12:32.703908 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.703848 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:12:32.703908 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.703897 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:12:32.919354 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.919297 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" event={"ID":"29bc04a6-3a7f-4574-9017-ead0d778ac34","Type":"ContainerStarted","Data":"b7a5b79af4a9b95a19edf48339148ac4a6312fb77a98ed43c2a0c6aa8a53d131"} Nov 02 18:12:32.919553 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.919380 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" event={"ID":"29bc04a6-3a7f-4574-9017-ead0d778ac34","Type":"ContainerStarted","Data":"99421d3b665e0b338797f494bcc057603b825005e726de8c5b84d91147d37ccc"} Nov 02 18:12:32.969782 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:32.969649 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" podStartSLOduration=1.9696175280000001 podStartE2EDuration="1.969617528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:12:32.968836341 +0000 UTC m=+2570.664191813" watchObservedRunningTime="2026-11-02 18:12:32.969617528 +0000 UTC m=+2570.664973002" Nov 02 18:12:33.335592 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:33.335555 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:33.337086 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:33.337056 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Nov 02 18:12:33.337207 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:33.337107 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" podUID="29bc04a6-3a7f-4574-9017-ead0d778ac34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Nov 02 18:12:34.335657 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:34.335618 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Nov 02 18:12:34.336178 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:34.335696 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" podUID="29bc04a6-3a7f-4574-9017-ead0d778ac34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Nov 02 18:12:35.043160 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:35.043126 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx"] Nov 02 18:12:35.043684 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:35.043394 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" podUID="e5bb3fbe-93ce-44aa-9cf1-02394cc28af7" containerName="istio-proxy" containerID="cri-o://f7e74a5eade9d1084418f6b7343fd52192b0e9e291fbf6b5293314890e6f55d4" gracePeriod=30 Nov 02 18:12:35.334936 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:35.334860 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Nov 02 18:12:35.334936 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:35.334914 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" podUID="29bc04a6-3a7f-4574-9017-ead0d778ac34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Nov 02 18:12:36.335161 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:36.335120 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Nov 02 18:12:36.335573 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:36.335191 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" podUID="29bc04a6-3a7f-4574-9017-ead0d778ac34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Nov 02 18:12:37.335995 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:37.335961 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Nov 02 18:12:37.336505 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:37.336021 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" podUID="29bc04a6-3a7f-4574-9017-ead0d778ac34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Nov 02 18:12:38.335263 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:38.335224 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Nov 02 18:12:38.335476 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:38.335287 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" podUID="29bc04a6-3a7f-4574-9017-ead0d778ac34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Nov 02 18:12:39.203907 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:12:39.203861 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:12:39.335164 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:39.335126 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Nov 02 18:12:39.335334 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:39.335212 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" podUID="29bc04a6-3a7f-4574-9017-ead0d778ac34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Nov 02 18:12:40.058857 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:40.058814 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": read tcp 10.133.0.2:35754->10.133.0.100:15021: read: connection reset by peer" start-of-body= Nov 02 18:12:40.059063 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:40.058900 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" podUID="e5bb3fbe-93ce-44aa-9cf1-02394cc28af7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": read tcp 10.133.0.2:35754->10.133.0.100:15021: read: connection reset by peer" Nov 02 18:12:40.335146 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:40.335116 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Nov 02 18:12:40.335533 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:40.335165 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" podUID="29bc04a6-3a7f-4574-9017-ead0d778ac34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Nov 02 18:12:40.348228 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:40.348201 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 18:12:40.476869 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:40.476836 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-credential-socket\") pod \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " Nov 02 18:12:40.477150 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:40.476889 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istio-envoy\") pod \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " Nov 02 18:12:40.477150 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:40.476952 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istio-podinfo\") pod \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " Nov 02 18:12:40.477150 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:40.477010 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-workload-certs\") pod \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " Nov 02 18:12:40.477150 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:40.477045 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-workload-socket\") pod \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " Nov 02 18:12:40.477150 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:40.477119 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istiod-ca-cert\") pod \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " Nov 02 18:12:40.477150 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:40.477122 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-credential-socket" pod "e5bb3fbe-93ce-44aa-9cf1-02394cc28af7" (UID: "e5bb3fbe-93ce-44aa-9cf1-02394cc28af7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:40.477519 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:40.477179 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-kube-api-access-sgmdr\" (UniqueName: \"kubernetes.io/projected/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-kube-api-access-sgmdr\") pod \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " Nov 02 18:12:40.477519 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:40.477222 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istio-data\") pod \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " Nov 02 18:12:40.477519 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:40.477288 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istio-token\" (UniqueName: \"kubernetes.io/projected/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istio-token\") pod \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\" (UID: \"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7\") " Nov 02 18:12:40.477698 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:40.477605 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-workload-socket" pod "e5bb3fbe-93ce-44aa-9cf1-02394cc28af7" (UID: "e5bb3fbe-93ce-44aa-9cf1-02394cc28af7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:40.477759 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:40.477693 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-workload-certs" pod "e5bb3fbe-93ce-44aa-9cf1-02394cc28af7" (UID: "e5bb3fbe-93ce-44aa-9cf1-02394cc28af7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:40.477759 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:40.477709 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istio-data" pod "e5bb3fbe-93ce-44aa-9cf1-02394cc28af7" (UID: "e5bb3fbe-93ce-44aa-9cf1-02394cc28af7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:40.477759 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:40.477734 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istiod-ca-cert" pod "e5bb3fbe-93ce-44aa-9cf1-02394cc28af7" (UID: "e5bb3fbe-93ce-44aa-9cf1-02394cc28af7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:40.477921 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:40.477879 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:40.477921 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:40.477901 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:40.477921 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:40.477917 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:40.478060 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:40.477931 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:40.478060 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:40.477944 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:40.479563 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:40.479529 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istio-podinfo" pod "e5bb3fbe-93ce-44aa-9cf1-02394cc28af7" (UID: "e5bb3fbe-93ce-44aa-9cf1-02394cc28af7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:12:40.479866 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:40.479848 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-kube-api-access-sgmdr" pod "e5bb3fbe-93ce-44aa-9cf1-02394cc28af7" (UID: "e5bb3fbe-93ce-44aa-9cf1-02394cc28af7"). InnerVolumeSpecName "kube-api-access-sgmdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:40.479948 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:40.479897 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istio-token" pod "e5bb3fbe-93ce-44aa-9cf1-02394cc28af7" (UID: "e5bb3fbe-93ce-44aa-9cf1-02394cc28af7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:40.480006 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:40.479991 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istio-envoy" pod "e5bb3fbe-93ce-44aa-9cf1-02394cc28af7" (UID: "e5bb3fbe-93ce-44aa-9cf1-02394cc28af7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:40.579768 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:40.579674 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgmdr\" (UniqueName: \"kubernetes.io/projected/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-kube-api-access-sgmdr\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:40.579768 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:40.579710 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:40.579768 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:40.579724 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:40.579768 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:40.579737 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:40.882990 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:40.882891 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8"] Nov 02 18:12:40.883187 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:40.883135 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" podUID="85212099-c279-459d-a468-b868bb55071b" containerName="istio-proxy" containerID="cri-o://4ca6c89eda89cedd6c28328c6c5825b82ec5d26142af68e6b0e01f39d21f6002" gracePeriod=30 Nov 02 18:12:41.095215 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:41.095172 4471 generic.go:356] "Generic (PLEG): container finished" podID="e5bb3fbe-93ce-44aa-9cf1-02394cc28af7" containerID="f7e74a5eade9d1084418f6b7343fd52192b0e9e291fbf6b5293314890e6f55d4" exitCode=0 Nov 02 18:12:41.095772 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:41.095736 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" event={"ID":"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7","Type":"ContainerDied","Data":"f7e74a5eade9d1084418f6b7343fd52192b0e9e291fbf6b5293314890e6f55d4"} Nov 02 18:12:41.095994 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:41.095974 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" event={"ID":"e5bb3fbe-93ce-44aa-9cf1-02394cc28af7","Type":"ContainerDied","Data":"706cdf45b1d340dc75ecbff3678f6258d9bbdd3173e2c274874d0e56855389b7"} Nov 02 18:12:41.096142 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:41.096126 4471 scope.go:122] "RemoveContainer" containerID="f7e74a5eade9d1084418f6b7343fd52192b0e9e291fbf6b5293314890e6f55d4" Nov 02 18:12:41.099158 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:41.099134 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx" Nov 02 18:12:41.127423 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:41.127390 4471 scope.go:122] "RemoveContainer" containerID="f7e74a5eade9d1084418f6b7343fd52192b0e9e291fbf6b5293314890e6f55d4" Nov 02 18:12:41.128065 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:12:41.128024 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7e74a5eade9d1084418f6b7343fd52192b0e9e291fbf6b5293314890e6f55d4\": container with ID starting with f7e74a5eade9d1084418f6b7343fd52192b0e9e291fbf6b5293314890e6f55d4 not found: ID does not exist" containerID="f7e74a5eade9d1084418f6b7343fd52192b0e9e291fbf6b5293314890e6f55d4" Nov 02 18:12:41.128161 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:41.128078 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7e74a5eade9d1084418f6b7343fd52192b0e9e291fbf6b5293314890e6f55d4"} err="failed to get container status \"f7e74a5eade9d1084418f6b7343fd52192b0e9e291fbf6b5293314890e6f55d4\": rpc error: code = NotFound desc = could not find container \"f7e74a5eade9d1084418f6b7343fd52192b0e9e291fbf6b5293314890e6f55d4\": container with ID starting with f7e74a5eade9d1084418f6b7343fd52192b0e9e291fbf6b5293314890e6f55d4 not found: ID does not exist" Nov 02 18:12:41.169178 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:41.169083 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx"] Nov 02 18:12:41.175709 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:41.175677 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-wdnxx"] Nov 02 18:12:41.339582 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:41.339549 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:41.340056 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:41.340033 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:41.340848 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:41.340828 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:41.456466 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:41.456382 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq"] Nov 02 18:12:42.105996 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:42.105964 4471 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" secret="" err="secret \"e2e-worker-w1-120394-openshift-default-dockercfg-qq496\" not found" Nov 02 18:12:42.441691 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:42.441620 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5bb3fbe-93ce-44aa-9cf1-02394cc28af7" path="/var/lib/kubelet/pods/e5bb3fbe-93ce-44aa-9cf1-02394cc28af7/volumes" Nov 02 18:12:43.112699 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:43.112657 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" podUID="29bc04a6-3a7f-4574-9017-ead0d778ac34" containerName="istio-proxy" containerID="cri-o://b7a5b79af4a9b95a19edf48339148ac4a6312fb77a98ed43c2a0c6aa8a53d131" gracePeriod=30 Nov 02 18:12:46.097654 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.097614 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-f6da5f-57dbdd6f67-xsfnw"] Nov 02 18:12:46.133189 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.133159 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f6da5f-6cb75c9495-682jg"] Nov 02 18:12:46.135151 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.135032 4471 generic.go:356] "Generic (PLEG): container finished" podID="85212099-c279-459d-a468-b868bb55071b" containerID="4ca6c89eda89cedd6c28328c6c5825b82ec5d26142af68e6b0e01f39d21f6002" exitCode=0 Nov 02 18:12:46.135151 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.135097 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" event={"ID":"85212099-c279-459d-a468-b868bb55071b","Type":"ContainerDied","Data":"4ca6c89eda89cedd6c28328c6c5825b82ec5d26142af68e6b0e01f39d21f6002"} Nov 02 18:12:46.135151 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.135144 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" event={"ID":"85212099-c279-459d-a468-b868bb55071b","Type":"ContainerDied","Data":"ee6c145baa206bc871ede8a942207d664e8c1dd33bd414ead0d309f25340aeac"} Nov 02 18:12:46.135425 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.135161 4471 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee6c145baa206bc871ede8a942207d664e8c1dd33bd414ead0d309f25340aeac" Nov 02 18:12:46.135425 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.135174 4471 scope.go:122] "RemoveContainer" containerID="8823091ca9765a3541591ff5624cda5fc772051fd423969892c219e35f7bfb6e" Nov 02 18:12:46.158512 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.158490 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 18:12:46.192725 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.192690 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-credential-socket\") pod \"85212099-c279-459d-a468-b868bb55071b\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " Nov 02 18:12:46.192918 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.192768 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85212099-c279-459d-a468-b868bb55071b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85212099-c279-459d-a468-b868bb55071b-istiod-ca-cert\") pod \"85212099-c279-459d-a468-b868bb55071b\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " Nov 02 18:12:46.192918 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.192836 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85212099-c279-459d-a468-b868bb55071b-istio-token\" (UniqueName: \"kubernetes.io/projected/85212099-c279-459d-a468-b868bb55071b-istio-token\") pod \"85212099-c279-459d-a468-b868bb55071b\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " Nov 02 18:12:46.193035 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.192944 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85212099-c279-459d-a468-b868bb55071b-kube-api-access-46xsg\" (UniqueName: \"kubernetes.io/projected/85212099-c279-459d-a468-b868bb55071b-kube-api-access-46xsg\") pod \"85212099-c279-459d-a468-b868bb55071b\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " Nov 02 18:12:46.193035 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.192996 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-istio-envoy\") pod \"85212099-c279-459d-a468-b868bb55071b\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " Nov 02 18:12:46.193146 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.193066 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-istio-data\") pod \"85212099-c279-459d-a468-b868bb55071b\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " Nov 02 18:12:46.193146 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.193073 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-credential-socket" pod "85212099-c279-459d-a468-b868bb55071b" (UID: "85212099-c279-459d-a468-b868bb55071b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:46.193146 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.193120 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85212099-c279-459d-a468-b868bb55071b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85212099-c279-459d-a468-b868bb55071b-istio-podinfo\") pod \"85212099-c279-459d-a468-b868bb55071b\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " Nov 02 18:12:46.193335 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.193158 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-workload-socket\") pod \"85212099-c279-459d-a468-b868bb55071b\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " Nov 02 18:12:46.193335 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.193253 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-workload-certs\") pod \"85212099-c279-459d-a468-b868bb55071b\" (UID: \"85212099-c279-459d-a468-b868bb55071b\") " Nov 02 18:12:46.193459 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.193331 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-istio-data" pod "85212099-c279-459d-a468-b868bb55071b" (UID: "85212099-c279-459d-a468-b868bb55071b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:46.193845 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.193818 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-workload-socket" pod "85212099-c279-459d-a468-b868bb55071b" (UID: "85212099-c279-459d-a468-b868bb55071b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:46.193986 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.193873 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:46.193986 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.193896 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:46.194296 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.194274 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-workload-certs" pod "85212099-c279-459d-a468-b868bb55071b" (UID: "85212099-c279-459d-a468-b868bb55071b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:46.194473 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.194450 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85212099-c279-459d-a468-b868bb55071b-istiod-ca-cert" pod "85212099-c279-459d-a468-b868bb55071b" (UID: "85212099-c279-459d-a468-b868bb55071b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:46.196387 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.196348 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85212099-c279-459d-a468-b868bb55071b-istio-token" pod "85212099-c279-459d-a468-b868bb55071b" (UID: "85212099-c279-459d-a468-b868bb55071b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:46.196629 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.196603 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85212099-c279-459d-a468-b868bb55071b-kube-api-access-46xsg" pod "85212099-c279-459d-a468-b868bb55071b" (UID: "85212099-c279-459d-a468-b868bb55071b"). InnerVolumeSpecName "kube-api-access-46xsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:46.196774 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.196754 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85212099-c279-459d-a468-b868bb55071b-istio-podinfo" pod "85212099-c279-459d-a468-b868bb55071b" (UID: "85212099-c279-459d-a468-b868bb55071b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:12:46.197755 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.197732 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-istio-envoy" pod "85212099-c279-459d-a468-b868bb55071b" (UID: "85212099-c279-459d-a468-b868bb55071b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:46.294905 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.294871 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85212099-c279-459d-a468-b868bb55071b-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:46.294905 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.294903 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46xsg\" (UniqueName: \"kubernetes.io/projected/85212099-c279-459d-a468-b868bb55071b-kube-api-access-46xsg\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:46.294905 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.294912 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:46.295149 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.294921 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85212099-c279-459d-a468-b868bb55071b-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:46.295149 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.294929 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:46.295149 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.294936 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85212099-c279-459d-a468-b868bb55071b-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:46.295149 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:46.294944 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85212099-c279-459d-a468-b868bb55071b-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:47.143181 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:47.143152 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8" Nov 02 18:12:47.245363 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:47.245326 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8"] Nov 02 18:12:47.249392 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:47.249366 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f59572-openshift-default-55d5967895-fd5d8"] Nov 02 18:12:48.364615 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:48.364585 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:48.420300 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:48.420219 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-istio-data\") pod \"29bc04a6-3a7f-4574-9017-ead0d778ac34\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " Nov 02 18:12:48.420300 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:48.420270 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-workload-certs\") pod \"29bc04a6-3a7f-4574-9017-ead0d778ac34\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " Nov 02 18:12:48.420565 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:48.420358 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-credential-socket\") pod \"29bc04a6-3a7f-4574-9017-ead0d778ac34\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " Nov 02 18:12:48.420565 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:48.420417 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29bc04a6-3a7f-4574-9017-ead0d778ac34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29bc04a6-3a7f-4574-9017-ead0d778ac34-istiod-ca-cert\") pod \"29bc04a6-3a7f-4574-9017-ead0d778ac34\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " Nov 02 18:12:48.420565 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:48.420506 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29bc04a6-3a7f-4574-9017-ead0d778ac34-istio-token\" (UniqueName: \"kubernetes.io/projected/29bc04a6-3a7f-4574-9017-ead0d778ac34-istio-token\") pod \"29bc04a6-3a7f-4574-9017-ead0d778ac34\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " Nov 02 18:12:48.420565 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:48.420542 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-istio-envoy\") pod \"29bc04a6-3a7f-4574-9017-ead0d778ac34\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " Nov 02 18:12:48.420796 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:48.420592 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-workload-certs" pod "29bc04a6-3a7f-4574-9017-ead0d778ac34" (UID: "29bc04a6-3a7f-4574-9017-ead0d778ac34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:48.420796 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:48.420607 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-istio-data" pod "29bc04a6-3a7f-4574-9017-ead0d778ac34" (UID: "29bc04a6-3a7f-4574-9017-ead0d778ac34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:48.420796 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:48.420653 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29bc04a6-3a7f-4574-9017-ead0d778ac34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29bc04a6-3a7f-4574-9017-ead0d778ac34-istio-podinfo\") pod \"29bc04a6-3a7f-4574-9017-ead0d778ac34\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " Nov 02 18:12:48.420796 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:48.420685 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-credential-socket" pod "29bc04a6-3a7f-4574-9017-ead0d778ac34" (UID: "29bc04a6-3a7f-4574-9017-ead0d778ac34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:48.420796 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:48.420699 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-workload-socket\") pod \"29bc04a6-3a7f-4574-9017-ead0d778ac34\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " Nov 02 18:12:48.421194 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:48.420857 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29bc04a6-3a7f-4574-9017-ead0d778ac34-kube-api-access-kg9d9\" (UniqueName: \"kubernetes.io/projected/29bc04a6-3a7f-4574-9017-ead0d778ac34-kube-api-access-kg9d9\") pod \"29bc04a6-3a7f-4574-9017-ead0d778ac34\" (UID: \"29bc04a6-3a7f-4574-9017-ead0d778ac34\") " Nov 02 18:12:48.421194 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:48.420873 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-workload-socket" pod "29bc04a6-3a7f-4574-9017-ead0d778ac34" (UID: "29bc04a6-3a7f-4574-9017-ead0d778ac34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:48.421194 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:48.420921 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29bc04a6-3a7f-4574-9017-ead0d778ac34-istiod-ca-cert" pod "29bc04a6-3a7f-4574-9017-ead0d778ac34" (UID: "29bc04a6-3a7f-4574-9017-ead0d778ac34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:48.421384 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:48.421330 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29bc04a6-3a7f-4574-9017-ead0d778ac34-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:48.421384 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:48.421354 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:48.421384 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:48.421367 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:48.421384 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:48.421376 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:48.421384 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:48.421384 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:48.423457 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:48.423246 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29bc04a6-3a7f-4574-9017-ead0d778ac34-istio-podinfo" pod "29bc04a6-3a7f-4574-9017-ead0d778ac34" (UID: "29bc04a6-3a7f-4574-9017-ead0d778ac34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:12:48.428563 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:48.424182 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29bc04a6-3a7f-4574-9017-ead0d778ac34-istio-token" pod "29bc04a6-3a7f-4574-9017-ead0d778ac34" (UID: "29bc04a6-3a7f-4574-9017-ead0d778ac34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:48.428563 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:48.424430 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-istio-envoy" pod "29bc04a6-3a7f-4574-9017-ead0d778ac34" (UID: "29bc04a6-3a7f-4574-9017-ead0d778ac34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:48.428563 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:48.424564 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29bc04a6-3a7f-4574-9017-ead0d778ac34-kube-api-access-kg9d9" pod "29bc04a6-3a7f-4574-9017-ead0d778ac34" (UID: "29bc04a6-3a7f-4574-9017-ead0d778ac34"). InnerVolumeSpecName "kube-api-access-kg9d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:48.443893 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:48.443864 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85212099-c279-459d-a468-b868bb55071b" path="/var/lib/kubelet/pods/85212099-c279-459d-a468-b868bb55071b/volumes" Nov 02 18:12:48.522618 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:48.522586 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29bc04a6-3a7f-4574-9017-ead0d778ac34-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:48.522618 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:48.522610 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29bc04a6-3a7f-4574-9017-ead0d778ac34-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:48.522618 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:48.522619 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29bc04a6-3a7f-4574-9017-ead0d778ac34-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:48.522854 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:48.522627 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg9d9\" (UniqueName: \"kubernetes.io/projected/29bc04a6-3a7f-4574-9017-ead0d778ac34-kube-api-access-kg9d9\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:49.157941 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:49.157906 4471 generic.go:356] "Generic (PLEG): container finished" podID="29bc04a6-3a7f-4574-9017-ead0d778ac34" containerID="b7a5b79af4a9b95a19edf48339148ac4a6312fb77a98ed43c2a0c6aa8a53d131" exitCode=0 Nov 02 18:12:49.158149 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:49.157961 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" Nov 02 18:12:49.158149 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:49.157990 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" event={"ID":"29bc04a6-3a7f-4574-9017-ead0d778ac34","Type":"ContainerDied","Data":"b7a5b79af4a9b95a19edf48339148ac4a6312fb77a98ed43c2a0c6aa8a53d131"} Nov 02 18:12:49.158149 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:49.158046 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq" event={"ID":"29bc04a6-3a7f-4574-9017-ead0d778ac34","Type":"ContainerDied","Data":"99421d3b665e0b338797f494bcc057603b825005e726de8c5b84d91147d37ccc"} Nov 02 18:12:49.158149 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:49.158070 4471 scope.go:122] "RemoveContainer" containerID="b7a5b79af4a9b95a19edf48339148ac4a6312fb77a98ed43c2a0c6aa8a53d131" Nov 02 18:12:49.171125 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:49.171107 4471 scope.go:122] "RemoveContainer" containerID="b7a5b79af4a9b95a19edf48339148ac4a6312fb77a98ed43c2a0c6aa8a53d131" Nov 02 18:12:49.171437 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:12:49.171411 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7a5b79af4a9b95a19edf48339148ac4a6312fb77a98ed43c2a0c6aa8a53d131\": container with ID starting with b7a5b79af4a9b95a19edf48339148ac4a6312fb77a98ed43c2a0c6aa8a53d131 not found: ID does not exist" containerID="b7a5b79af4a9b95a19edf48339148ac4a6312fb77a98ed43c2a0c6aa8a53d131" Nov 02 18:12:49.171556 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:49.171442 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7a5b79af4a9b95a19edf48339148ac4a6312fb77a98ed43c2a0c6aa8a53d131"} err="failed to get container status \"b7a5b79af4a9b95a19edf48339148ac4a6312fb77a98ed43c2a0c6aa8a53d131\": rpc error: code = NotFound desc = could not find container \"b7a5b79af4a9b95a19edf48339148ac4a6312fb77a98ed43c2a0c6aa8a53d131\": container with ID starting with b7a5b79af4a9b95a19edf48339148ac4a6312fb77a98ed43c2a0c6aa8a53d131 not found: ID does not exist" Nov 02 18:12:49.225242 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:49.225208 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq"] Nov 02 18:12:49.238592 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:49.238557 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-rctxq"] Nov 02 18:12:50.449403 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:50.449366 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29bc04a6-3a7f-4574-9017-ead0d778ac34" path="/var/lib/kubelet/pods/29bc04a6-3a7f-4574-9017-ead0d778ac34/volumes" Nov 02 18:12:51.098586 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:51.098531 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f6da5f-57dbdd6f67-xsfnw" podUID="00044b6e-d520-4934-ba8a-b2d638f899d6" containerName="payload-processing" containerID="cri-o://1a824126f4408c3860bd740bfa885601d050dd42ba4568c543b51f91c1ddddf0" gracePeriod=25 Nov 02 18:12:51.133695 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:51.133655 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f6da5f-6cb75c9495-682jg" podUID="d2d5291d-d4e9-462b-8f3e-f87f3b82d946" containerName="payload-pre-processing" containerID="cri-o://8ad5d40ad115398ef98da8a1639a5c44dde7465d108d0f715edb66c7188bfbe0" gracePeriod=25 Nov 02 18:12:51.448568 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:51.448539 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f6da5f-57dbdd6f67-xsfnw" Nov 02 18:12:51.451560 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:51.451538 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f6da5f-6cb75c9495-682jg" Nov 02 18:12:51.561741 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:51.561705 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00044b6e-d520-4934-ba8a-b2d638f899d6-kube-api-access-zshjm\" (UniqueName: \"kubernetes.io/projected/00044b6e-d520-4934-ba8a-b2d638f899d6-kube-api-access-zshjm\") pod \"00044b6e-d520-4934-ba8a-b2d638f899d6\" (UID: \"00044b6e-d520-4934-ba8a-b2d638f899d6\") " Nov 02 18:12:51.561971 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:51.561814 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00044b6e-d520-4934-ba8a-b2d638f899d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00044b6e-d520-4934-ba8a-b2d638f899d6-plugins-config-volume\") pod \"00044b6e-d520-4934-ba8a-b2d638f899d6\" (UID: \"00044b6e-d520-4934-ba8a-b2d638f899d6\") " Nov 02 18:12:51.561971 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:51.561925 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2d5291d-d4e9-462b-8f3e-f87f3b82d946-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d5291d-d4e9-462b-8f3e-f87f3b82d946-plugins-config-volume\") pod \"d2d5291d-d4e9-462b-8f3e-f87f3b82d946\" (UID: \"d2d5291d-d4e9-462b-8f3e-f87f3b82d946\") " Nov 02 18:12:51.562104 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:51.561992 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2d5291d-d4e9-462b-8f3e-f87f3b82d946-kube-api-access-7kqx9\" (UniqueName: \"kubernetes.io/projected/d2d5291d-d4e9-462b-8f3e-f87f3b82d946-kube-api-access-7kqx9\") pod \"d2d5291d-d4e9-462b-8f3e-f87f3b82d946\" (UID: \"d2d5291d-d4e9-462b-8f3e-f87f3b82d946\") " Nov 02 18:12:51.562262 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:51.562220 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00044b6e-d520-4934-ba8a-b2d638f899d6-plugins-config-volume" pod "00044b6e-d520-4934-ba8a-b2d638f899d6" (UID: "00044b6e-d520-4934-ba8a-b2d638f899d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:51.562416 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:51.562389 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2d5291d-d4e9-462b-8f3e-f87f3b82d946-plugins-config-volume" pod "d2d5291d-d4e9-462b-8f3e-f87f3b82d946" (UID: "d2d5291d-d4e9-462b-8f3e-f87f3b82d946"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:51.562601 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:51.562575 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00044b6e-d520-4934-ba8a-b2d638f899d6-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:51.562681 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:51.562603 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d5291d-d4e9-462b-8f3e-f87f3b82d946-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:51.564371 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:51.564345 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2d5291d-d4e9-462b-8f3e-f87f3b82d946-kube-api-access-7kqx9" pod "d2d5291d-d4e9-462b-8f3e-f87f3b82d946" (UID: "d2d5291d-d4e9-462b-8f3e-f87f3b82d946"). InnerVolumeSpecName "kube-api-access-7kqx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:51.564549 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:51.564533 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00044b6e-d520-4934-ba8a-b2d638f899d6-kube-api-access-zshjm" pod "00044b6e-d520-4934-ba8a-b2d638f899d6" (UID: "00044b6e-d520-4934-ba8a-b2d638f899d6"). InnerVolumeSpecName "kube-api-access-zshjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:51.664009 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:51.663930 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kqx9\" (UniqueName: \"kubernetes.io/projected/d2d5291d-d4e9-462b-8f3e-f87f3b82d946-kube-api-access-7kqx9\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:51.664009 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:51.663959 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zshjm\" (UniqueName: \"kubernetes.io/projected/00044b6e-d520-4934-ba8a-b2d638f899d6-kube-api-access-zshjm\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:51.949274 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:51.949185 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-2672ed-69855f494c-rpbz7"] Nov 02 18:12:51.984703 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:51.984668 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2672ed-747f99c7d7-mm9nh"] Nov 02 18:12:52.190827 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:52.190789 4471 generic.go:356] "Generic (PLEG): container finished" podID="00044b6e-d520-4934-ba8a-b2d638f899d6" containerID="1a824126f4408c3860bd740bfa885601d050dd42ba4568c543b51f91c1ddddf0" exitCode=0 Nov 02 18:12:52.199486 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:52.196005 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f6da5f-57dbdd6f67-xsfnw" Nov 02 18:12:52.199486 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:52.196034 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f6da5f-57dbdd6f67-xsfnw" event={"ID":"00044b6e-d520-4934-ba8a-b2d638f899d6","Type":"ContainerDied","Data":"1a824126f4408c3860bd740bfa885601d050dd42ba4568c543b51f91c1ddddf0"} Nov 02 18:12:52.199486 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:52.199228 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f6da5f-57dbdd6f67-xsfnw" event={"ID":"00044b6e-d520-4934-ba8a-b2d638f899d6","Type":"ContainerDied","Data":"d2d51cbf029e2b8255890d635215abcfa30c793a1b1101f5a4b20fea9274236f"} Nov 02 18:12:52.199486 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:52.199255 4471 scope.go:122] "RemoveContainer" containerID="1a824126f4408c3860bd740bfa885601d050dd42ba4568c543b51f91c1ddddf0" Nov 02 18:12:52.201039 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:52.201011 4471 generic.go:356] "Generic (PLEG): container finished" podID="d2d5291d-d4e9-462b-8f3e-f87f3b82d946" containerID="8ad5d40ad115398ef98da8a1639a5c44dde7465d108d0f715edb66c7188bfbe0" exitCode=0 Nov 02 18:12:52.201165 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:52.201059 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f6da5f-6cb75c9495-682jg" event={"ID":"d2d5291d-d4e9-462b-8f3e-f87f3b82d946","Type":"ContainerDied","Data":"8ad5d40ad115398ef98da8a1639a5c44dde7465d108d0f715edb66c7188bfbe0"} Nov 02 18:12:52.201165 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:52.201066 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f6da5f-6cb75c9495-682jg" Nov 02 18:12:52.201165 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:52.201095 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f6da5f-6cb75c9495-682jg" event={"ID":"d2d5291d-d4e9-462b-8f3e-f87f3b82d946","Type":"ContainerDied","Data":"e6c8ec8ed16010d2393c12ff9e56db44449777a52e5c53a02998136891e76cbb"} Nov 02 18:12:52.212721 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:52.212702 4471 scope.go:122] "RemoveContainer" containerID="1a824126f4408c3860bd740bfa885601d050dd42ba4568c543b51f91c1ddddf0" Nov 02 18:12:52.213057 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:12:52.213032 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a824126f4408c3860bd740bfa885601d050dd42ba4568c543b51f91c1ddddf0\": container with ID starting with 1a824126f4408c3860bd740bfa885601d050dd42ba4568c543b51f91c1ddddf0 not found: ID does not exist" containerID="1a824126f4408c3860bd740bfa885601d050dd42ba4568c543b51f91c1ddddf0" Nov 02 18:12:52.213167 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:52.213068 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a824126f4408c3860bd740bfa885601d050dd42ba4568c543b51f91c1ddddf0"} err="failed to get container status \"1a824126f4408c3860bd740bfa885601d050dd42ba4568c543b51f91c1ddddf0\": rpc error: code = NotFound desc = could not find container \"1a824126f4408c3860bd740bfa885601d050dd42ba4568c543b51f91c1ddddf0\": container with ID starting with 1a824126f4408c3860bd740bfa885601d050dd42ba4568c543b51f91c1ddddf0 not found: ID does not exist" Nov 02 18:12:52.213167 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:52.213092 4471 scope.go:122] "RemoveContainer" containerID="8ad5d40ad115398ef98da8a1639a5c44dde7465d108d0f715edb66c7188bfbe0" Nov 02 18:12:52.223154 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:52.223134 4471 scope.go:122] "RemoveContainer" containerID="8ad5d40ad115398ef98da8a1639a5c44dde7465d108d0f715edb66c7188bfbe0" Nov 02 18:12:52.223465 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:12:52.223441 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ad5d40ad115398ef98da8a1639a5c44dde7465d108d0f715edb66c7188bfbe0\": container with ID starting with 8ad5d40ad115398ef98da8a1639a5c44dde7465d108d0f715edb66c7188bfbe0 not found: ID does not exist" containerID="8ad5d40ad115398ef98da8a1639a5c44dde7465d108d0f715edb66c7188bfbe0" Nov 02 18:12:52.223530 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:52.223472 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ad5d40ad115398ef98da8a1639a5c44dde7465d108d0f715edb66c7188bfbe0"} err="failed to get container status \"8ad5d40ad115398ef98da8a1639a5c44dde7465d108d0f715edb66c7188bfbe0\": rpc error: code = NotFound desc = could not find container \"8ad5d40ad115398ef98da8a1639a5c44dde7465d108d0f715edb66c7188bfbe0\": container with ID starting with 8ad5d40ad115398ef98da8a1639a5c44dde7465d108d0f715edb66c7188bfbe0 not found: ID does not exist" Nov 02 18:12:52.254349 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:52.254323 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f6da5f-6cb75c9495-682jg"] Nov 02 18:12:52.258462 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:52.258442 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f6da5f-6cb75c9495-682jg"] Nov 02 18:12:52.293888 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:52.293859 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-f6da5f-57dbdd6f67-xsfnw"] Nov 02 18:12:52.302216 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:52.302187 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-f6da5f-57dbdd6f67-xsfnw"] Nov 02 18:12:52.441704 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:52.441672 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00044b6e-d520-4934-ba8a-b2d638f899d6" path="/var/lib/kubelet/pods/00044b6e-d520-4934-ba8a-b2d638f899d6/volumes" Nov 02 18:12:52.442071 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:52.442057 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2d5291d-d4e9-462b-8f3e-f87f3b82d946" path="/var/lib/kubelet/pods/d2d5291d-d4e9-462b-8f3e-f87f3b82d946/volumes" Nov 02 18:12:53.188104 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:53.188071 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-c810ca-7c7f98d89-lpltl"] Nov 02 18:12:53.219798 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:53.219763 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-c810ca-5fbdb874f9-2284p"] Nov 02 18:12:56.894280 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:56.894242 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn"] Nov 02 18:12:56.894796 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:56.894486 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn" podUID="141bdec6-5264-4a27-87e4-699a5bdcf2d7" containerName="maas-api" containerID="cri-o://ced4e72a3b61abf8381d61a5e4e7dbed420f1aaaeb7b1aec227eaae7bb68398e" gracePeriod=30 Nov 02 18:12:56.950252 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:56.950202 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2672ed-69855f494c-rpbz7" podUID="0381ed09-c883-448a-baf2-c7cde6ed90ab" containerName="payload-processing" containerID="cri-o://a24f727007bdbf63fa47b09515fabd28a2fc5e6b81a634b7acaf960fd71af3a0" gracePeriod=25 Nov 02 18:12:56.985993 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:56.985722 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2672ed-747f99c7d7-mm9nh" podUID="1408655c-5efa-4883-b9d2-0b51327d40bb" containerName="payload-pre-processing" containerID="cri-o://900fb89e03912d8d0fcedd0773d6d19aa70f40a99544bee32b86f7c29df1e49c" gracePeriod=25 Nov 02 18:12:57.010404 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:57.010368 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn" podUID="141bdec6-5264-4a27-87e4-699a5bdcf2d7" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.105:8443/health\": dial tcp 10.133.0.105:8443: connect: connection refused" Nov 02 18:12:57.247174 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:57.246239 4471 generic.go:356] "Generic (PLEG): container finished" podID="0381ed09-c883-448a-baf2-c7cde6ed90ab" containerID="a24f727007bdbf63fa47b09515fabd28a2fc5e6b81a634b7acaf960fd71af3a0" exitCode=0 Nov 02 18:12:57.247174 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:57.246327 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2672ed-69855f494c-rpbz7" event={"ID":"0381ed09-c883-448a-baf2-c7cde6ed90ab","Type":"ContainerDied","Data":"a24f727007bdbf63fa47b09515fabd28a2fc5e6b81a634b7acaf960fd71af3a0"} Nov 02 18:12:57.248992 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:57.248947 4471 generic.go:356] "Generic (PLEG): container finished" podID="141bdec6-5264-4a27-87e4-699a5bdcf2d7" containerID="ced4e72a3b61abf8381d61a5e4e7dbed420f1aaaeb7b1aec227eaae7bb68398e" exitCode=0 Nov 02 18:12:57.249122 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:57.249069 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn" event={"ID":"141bdec6-5264-4a27-87e4-699a5bdcf2d7","Type":"ContainerDied","Data":"ced4e72a3b61abf8381d61a5e4e7dbed420f1aaaeb7b1aec227eaae7bb68398e"} Nov 02 18:12:57.251189 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:57.251148 4471 generic.go:356] "Generic (PLEG): container finished" podID="1408655c-5efa-4883-b9d2-0b51327d40bb" containerID="900fb89e03912d8d0fcedd0773d6d19aa70f40a99544bee32b86f7c29df1e49c" exitCode=0 Nov 02 18:12:57.251321 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:57.251263 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2672ed-747f99c7d7-mm9nh" event={"ID":"1408655c-5efa-4883-b9d2-0b51327d40bb","Type":"ContainerDied","Data":"900fb89e03912d8d0fcedd0773d6d19aa70f40a99544bee32b86f7c29df1e49c"} Nov 02 18:12:57.438944 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:57.435535 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2672ed-69855f494c-rpbz7" Nov 02 18:12:57.521850 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:57.521822 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn" Nov 02 18:12:57.532593 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:57.532572 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2672ed-747f99c7d7-mm9nh" Nov 02 18:12:57.552784 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:57.552759 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/141bdec6-5264-4a27-87e4-699a5bdcf2d7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/141bdec6-5264-4a27-87e4-699a5bdcf2d7-metrics-certs\") pod \"141bdec6-5264-4a27-87e4-699a5bdcf2d7\" (UID: \"141bdec6-5264-4a27-87e4-699a5bdcf2d7\") " Nov 02 18:12:57.552899 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:57.552803 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/141bdec6-5264-4a27-87e4-699a5bdcf2d7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/141bdec6-5264-4a27-87e4-699a5bdcf2d7-openshift-service-ca\") pod \"141bdec6-5264-4a27-87e4-699a5bdcf2d7\" (UID: \"141bdec6-5264-4a27-87e4-699a5bdcf2d7\") " Nov 02 18:12:57.552899 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:57.552827 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/141bdec6-5264-4a27-87e4-699a5bdcf2d7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/141bdec6-5264-4a27-87e4-699a5bdcf2d7-maas-api-tls\") pod \"141bdec6-5264-4a27-87e4-699a5bdcf2d7\" (UID: \"141bdec6-5264-4a27-87e4-699a5bdcf2d7\") " Nov 02 18:12:57.553037 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:57.552914 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/141bdec6-5264-4a27-87e4-699a5bdcf2d7-kube-api-access-qv67c\" (UniqueName: \"kubernetes.io/projected/141bdec6-5264-4a27-87e4-699a5bdcf2d7-kube-api-access-qv67c\") pod \"141bdec6-5264-4a27-87e4-699a5bdcf2d7\" (UID: \"141bdec6-5264-4a27-87e4-699a5bdcf2d7\") " Nov 02 18:12:57.553037 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:57.552961 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0381ed09-c883-448a-baf2-c7cde6ed90ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0381ed09-c883-448a-baf2-c7cde6ed90ab-plugins-config-volume\") pod \"0381ed09-c883-448a-baf2-c7cde6ed90ab\" (UID: \"0381ed09-c883-448a-baf2-c7cde6ed90ab\") " Nov 02 18:12:57.553157 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:57.553042 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0381ed09-c883-448a-baf2-c7cde6ed90ab-kube-api-access-r4f7g\" (UniqueName: \"kubernetes.io/projected/0381ed09-c883-448a-baf2-c7cde6ed90ab-kube-api-access-r4f7g\") pod \"0381ed09-c883-448a-baf2-c7cde6ed90ab\" (UID: \"0381ed09-c883-448a-baf2-c7cde6ed90ab\") " Nov 02 18:12:57.553157 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:57.553111 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/141bdec6-5264-4a27-87e4-699a5bdcf2d7-openshift-service-ca" pod "141bdec6-5264-4a27-87e4-699a5bdcf2d7" (UID: "141bdec6-5264-4a27-87e4-699a5bdcf2d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:57.553408 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:57.553380 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0381ed09-c883-448a-baf2-c7cde6ed90ab-plugins-config-volume" pod "0381ed09-c883-448a-baf2-c7cde6ed90ab" (UID: "0381ed09-c883-448a-baf2-c7cde6ed90ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:57.553604 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:57.553581 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0381ed09-c883-448a-baf2-c7cde6ed90ab-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:57.553691 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:57.553608 4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/141bdec6-5264-4a27-87e4-699a5bdcf2d7-openshift-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:57.554945 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:57.554923 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/141bdec6-5264-4a27-87e4-699a5bdcf2d7-maas-api-tls" pod "141bdec6-5264-4a27-87e4-699a5bdcf2d7" (UID: "141bdec6-5264-4a27-87e4-699a5bdcf2d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:12:57.555020 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:57.554985 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/141bdec6-5264-4a27-87e4-699a5bdcf2d7-metrics-certs" pod "141bdec6-5264-4a27-87e4-699a5bdcf2d7" (UID: "141bdec6-5264-4a27-87e4-699a5bdcf2d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:12:57.555068 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:57.555044 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0381ed09-c883-448a-baf2-c7cde6ed90ab-kube-api-access-r4f7g" pod "0381ed09-c883-448a-baf2-c7cde6ed90ab" (UID: "0381ed09-c883-448a-baf2-c7cde6ed90ab"). InnerVolumeSpecName "kube-api-access-r4f7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:57.555150 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:57.555136 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/141bdec6-5264-4a27-87e4-699a5bdcf2d7-kube-api-access-qv67c" pod "141bdec6-5264-4a27-87e4-699a5bdcf2d7" (UID: "141bdec6-5264-4a27-87e4-699a5bdcf2d7"). InnerVolumeSpecName "kube-api-access-qv67c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:57.654036 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:57.654005 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1408655c-5efa-4883-b9d2-0b51327d40bb-kube-api-access-sqf5k\" (UniqueName: \"kubernetes.io/projected/1408655c-5efa-4883-b9d2-0b51327d40bb-kube-api-access-sqf5k\") pod \"1408655c-5efa-4883-b9d2-0b51327d40bb\" (UID: \"1408655c-5efa-4883-b9d2-0b51327d40bb\") " Nov 02 18:12:57.654192 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:57.654071 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1408655c-5efa-4883-b9d2-0b51327d40bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1408655c-5efa-4883-b9d2-0b51327d40bb-plugins-config-volume\") pod \"1408655c-5efa-4883-b9d2-0b51327d40bb\" (UID: \"1408655c-5efa-4883-b9d2-0b51327d40bb\") " Nov 02 18:12:57.654476 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:57.654450 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1408655c-5efa-4883-b9d2-0b51327d40bb-plugins-config-volume" pod "1408655c-5efa-4883-b9d2-0b51327d40bb" (UID: "1408655c-5efa-4883-b9d2-0b51327d40bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:57.654476 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:57.654469 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4f7g\" (UniqueName: \"kubernetes.io/projected/0381ed09-c883-448a-baf2-c7cde6ed90ab-kube-api-access-r4f7g\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:57.654610 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:57.654484 4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/141bdec6-5264-4a27-87e4-699a5bdcf2d7-metrics-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:57.654610 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:57.654492 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/141bdec6-5264-4a27-87e4-699a5bdcf2d7-maas-api-tls\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:57.654610 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:57.654500 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv67c\" (UniqueName: \"kubernetes.io/projected/141bdec6-5264-4a27-87e4-699a5bdcf2d7-kube-api-access-qv67c\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:57.681437 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:57.681340 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1408655c-5efa-4883-b9d2-0b51327d40bb-kube-api-access-sqf5k" pod "1408655c-5efa-4883-b9d2-0b51327d40bb" (UID: "1408655c-5efa-4883-b9d2-0b51327d40bb"). InnerVolumeSpecName "kube-api-access-sqf5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:57.761329 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:57.757226 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1408655c-5efa-4883-b9d2-0b51327d40bb-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:57.761329 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:57.757272 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqf5k\" (UniqueName: \"kubernetes.io/projected/1408655c-5efa-4883-b9d2-0b51327d40bb-kube-api-access-sqf5k\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:58.189375 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:58.189279 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-c810ca-7c7f98d89-lpltl" podUID="c3f2ca77-1f97-44a4-99f1-291daab2dee3" containerName="payload-processing" containerID="cri-o://1ee87bfed553dc07f5edc75837272fe91b5c1c1e4a0cea934927cfbc2808dbcf" gracePeriod=25 Nov 02 18:12:58.221008 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:58.220936 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-c810ca-5fbdb874f9-2284p" podUID="7395fee0-6fb5-4a90-be40-ba36fde4b163" containerName="payload-pre-processing" containerID="cri-o://cb0c675689427d74b5951cd9f40b0265a13da471912904e4f01754fe930abe98" gracePeriod=25 Nov 02 18:12:58.285009 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:58.284814 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2672ed-69855f494c-rpbz7" event={"ID":"0381ed09-c883-448a-baf2-c7cde6ed90ab","Type":"ContainerDied","Data":"cff9d81475896af4275fe9f88b4bb11cb1315ecd845dbf91bfe7f30b43cdcbce"} Nov 02 18:12:58.285009 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:58.284842 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2672ed-69855f494c-rpbz7" Nov 02 18:12:58.285009 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:58.284934 4471 scope.go:122] "RemoveContainer" containerID="a24f727007bdbf63fa47b09515fabd28a2fc5e6b81a634b7acaf960fd71af3a0" Nov 02 18:12:58.287572 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:58.287552 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn" Nov 02 18:12:58.287680 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:58.287573 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn" event={"ID":"141bdec6-5264-4a27-87e4-699a5bdcf2d7","Type":"ContainerDied","Data":"d46b239a618381b5dead56422b83078552f9c1d6633e857be018a064b60ca18b"} Nov 02 18:12:58.289132 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:58.289106 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2672ed-747f99c7d7-mm9nh" Nov 02 18:12:58.289236 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:58.289155 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2672ed-747f99c7d7-mm9nh" event={"ID":"1408655c-5efa-4883-b9d2-0b51327d40bb","Type":"ContainerDied","Data":"3eb7736cf7b57a5f4634b1ad38d9444d116acdc2da4f97a7127a73f08d1ee3a3"} Nov 02 18:12:58.335972 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:58.335943 4471 scope.go:122] "RemoveContainer" containerID="ced4e72a3b61abf8381d61a5e4e7dbed420f1aaaeb7b1aec227eaae7bb68398e" Nov 02 18:12:58.345934 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:58.345911 4471 scope.go:122] "RemoveContainer" containerID="900fb89e03912d8d0fcedd0773d6d19aa70f40a99544bee32b86f7c29df1e49c" Nov 02 18:12:58.409674 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:58.409638 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn"] Nov 02 18:12:58.429469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:58.429436 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f6da5f-85b5667f55-4z6jn"] Nov 02 18:12:58.445105 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:58.443223 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="141bdec6-5264-4a27-87e4-699a5bdcf2d7" path="/var/lib/kubelet/pods/141bdec6-5264-4a27-87e4-699a5bdcf2d7/volumes" Nov 02 18:12:58.452529 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:58.452502 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2672ed-747f99c7d7-mm9nh"] Nov 02 18:12:58.484508 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:58.484400 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2672ed-747f99c7d7-mm9nh"] Nov 02 18:12:58.527206 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:58.526982 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-2672ed-69855f494c-rpbz7"] Nov 02 18:12:58.530073 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:58.530045 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-2672ed-69855f494c-rpbz7"] Nov 02 18:12:58.542380 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:58.542357 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-c810ca-7c7f98d89-lpltl" Nov 02 18:12:58.591351 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:58.591323 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-c810ca-5fbdb874f9-2284p" Nov 02 18:12:58.687698 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:58.687665 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3f2ca77-1f97-44a4-99f1-291daab2dee3-kube-api-access-w46g8\" (UniqueName: \"kubernetes.io/projected/c3f2ca77-1f97-44a4-99f1-291daab2dee3-kube-api-access-w46g8\") pod \"c3f2ca77-1f97-44a4-99f1-291daab2dee3\" (UID: \"c3f2ca77-1f97-44a4-99f1-291daab2dee3\") " Nov 02 18:12:58.687698 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:58.687709 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7395fee0-6fb5-4a90-be40-ba36fde4b163-kube-api-access-dbww9\" (UniqueName: \"kubernetes.io/projected/7395fee0-6fb5-4a90-be40-ba36fde4b163-kube-api-access-dbww9\") pod \"7395fee0-6fb5-4a90-be40-ba36fde4b163\" (UID: \"7395fee0-6fb5-4a90-be40-ba36fde4b163\") " Nov 02 18:12:58.687959 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:58.687772 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7395fee0-6fb5-4a90-be40-ba36fde4b163-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7395fee0-6fb5-4a90-be40-ba36fde4b163-plugins-config-volume\") pod \"7395fee0-6fb5-4a90-be40-ba36fde4b163\" (UID: \"7395fee0-6fb5-4a90-be40-ba36fde4b163\") " Nov 02 18:12:58.687959 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:58.687829 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3f2ca77-1f97-44a4-99f1-291daab2dee3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3f2ca77-1f97-44a4-99f1-291daab2dee3-plugins-config-volume\") pod \"c3f2ca77-1f97-44a4-99f1-291daab2dee3\" (UID: \"c3f2ca77-1f97-44a4-99f1-291daab2dee3\") " Nov 02 18:12:58.688233 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:58.688206 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7395fee0-6fb5-4a90-be40-ba36fde4b163-plugins-config-volume" pod "7395fee0-6fb5-4a90-be40-ba36fde4b163" (UID: "7395fee0-6fb5-4a90-be40-ba36fde4b163"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:58.688348 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:58.688230 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3f2ca77-1f97-44a4-99f1-291daab2dee3-plugins-config-volume" pod "c3f2ca77-1f97-44a4-99f1-291daab2dee3" (UID: "c3f2ca77-1f97-44a4-99f1-291daab2dee3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:58.688430 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:58.688410 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3f2ca77-1f97-44a4-99f1-291daab2dee3-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:58.688476 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:58.688432 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7395fee0-6fb5-4a90-be40-ba36fde4b163-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:58.690400 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:58.690376 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3f2ca77-1f97-44a4-99f1-291daab2dee3-kube-api-access-w46g8" pod "c3f2ca77-1f97-44a4-99f1-291daab2dee3" (UID: "c3f2ca77-1f97-44a4-99f1-291daab2dee3"). InnerVolumeSpecName "kube-api-access-w46g8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:58.690400 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:58.690392 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7395fee0-6fb5-4a90-be40-ba36fde4b163-kube-api-access-dbww9" pod "7395fee0-6fb5-4a90-be40-ba36fde4b163" (UID: "7395fee0-6fb5-4a90-be40-ba36fde4b163"). InnerVolumeSpecName "kube-api-access-dbww9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:58.789691 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:58.789654 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w46g8\" (UniqueName: \"kubernetes.io/projected/c3f2ca77-1f97-44a4-99f1-291daab2dee3-kube-api-access-w46g8\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:58.789691 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:58.789690 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbww9\" (UniqueName: \"kubernetes.io/projected/7395fee0-6fb5-4a90-be40-ba36fde4b163-kube-api-access-dbww9\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:59.298271 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:59.298235 4471 generic.go:356] "Generic (PLEG): container finished" podID="7395fee0-6fb5-4a90-be40-ba36fde4b163" containerID="cb0c675689427d74b5951cd9f40b0265a13da471912904e4f01754fe930abe98" exitCode=0 Nov 02 18:12:59.298780 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:59.298298 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-c810ca-5fbdb874f9-2284p" Nov 02 18:12:59.298780 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:59.298330 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-c810ca-5fbdb874f9-2284p" event={"ID":"7395fee0-6fb5-4a90-be40-ba36fde4b163","Type":"ContainerDied","Data":"cb0c675689427d74b5951cd9f40b0265a13da471912904e4f01754fe930abe98"} Nov 02 18:12:59.298780 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:59.298393 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-c810ca-5fbdb874f9-2284p" event={"ID":"7395fee0-6fb5-4a90-be40-ba36fde4b163","Type":"ContainerDied","Data":"bf902f49b92da23239e02827fc2596223836f72f2fc0ef5f18c9cfa07292f662"} Nov 02 18:12:59.298780 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:59.298442 4471 scope.go:122] "RemoveContainer" containerID="cb0c675689427d74b5951cd9f40b0265a13da471912904e4f01754fe930abe98" Nov 02 18:12:59.300812 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:59.300786 4471 generic.go:356] "Generic (PLEG): container finished" podID="c3f2ca77-1f97-44a4-99f1-291daab2dee3" containerID="1ee87bfed553dc07f5edc75837272fe91b5c1c1e4a0cea934927cfbc2808dbcf" exitCode=0 Nov 02 18:12:59.300956 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:59.300846 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-c810ca-7c7f98d89-lpltl" Nov 02 18:12:59.300956 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:59.300887 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-c810ca-7c7f98d89-lpltl" event={"ID":"c3f2ca77-1f97-44a4-99f1-291daab2dee3","Type":"ContainerDied","Data":"1ee87bfed553dc07f5edc75837272fe91b5c1c1e4a0cea934927cfbc2808dbcf"} Nov 02 18:12:59.301086 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:59.300973 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-c810ca-7c7f98d89-lpltl" event={"ID":"c3f2ca77-1f97-44a4-99f1-291daab2dee3","Type":"ContainerDied","Data":"18c1810bfd5281493597fd9e35eef6dddd43bb0a18d4f08dce11f841db3de59d"} Nov 02 18:12:59.312896 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:59.312876 4471 scope.go:122] "RemoveContainer" containerID="cb0c675689427d74b5951cd9f40b0265a13da471912904e4f01754fe930abe98" Nov 02 18:12:59.313456 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:12:59.313432 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb0c675689427d74b5951cd9f40b0265a13da471912904e4f01754fe930abe98\": container with ID starting with cb0c675689427d74b5951cd9f40b0265a13da471912904e4f01754fe930abe98 not found: ID does not exist" containerID="cb0c675689427d74b5951cd9f40b0265a13da471912904e4f01754fe930abe98" Nov 02 18:12:59.313609 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:59.313577 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb0c675689427d74b5951cd9f40b0265a13da471912904e4f01754fe930abe98"} err="failed to get container status \"cb0c675689427d74b5951cd9f40b0265a13da471912904e4f01754fe930abe98\": rpc error: code = NotFound desc = could not find container \"cb0c675689427d74b5951cd9f40b0265a13da471912904e4f01754fe930abe98\": container with ID starting with cb0c675689427d74b5951cd9f40b0265a13da471912904e4f01754fe930abe98 not found: ID does not exist" Nov 02 18:12:59.313993 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:59.313609 4471 scope.go:122] "RemoveContainer" containerID="1ee87bfed553dc07f5edc75837272fe91b5c1c1e4a0cea934927cfbc2808dbcf" Nov 02 18:12:59.338290 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:59.338269 4471 scope.go:122] "RemoveContainer" containerID="1ee87bfed553dc07f5edc75837272fe91b5c1c1e4a0cea934927cfbc2808dbcf" Nov 02 18:12:59.338631 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:12:59.338597 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ee87bfed553dc07f5edc75837272fe91b5c1c1e4a0cea934927cfbc2808dbcf\": container with ID starting with 1ee87bfed553dc07f5edc75837272fe91b5c1c1e4a0cea934927cfbc2808dbcf not found: ID does not exist" containerID="1ee87bfed553dc07f5edc75837272fe91b5c1c1e4a0cea934927cfbc2808dbcf" Nov 02 18:12:59.338687 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:59.338644 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ee87bfed553dc07f5edc75837272fe91b5c1c1e4a0cea934927cfbc2808dbcf"} err="failed to get container status \"1ee87bfed553dc07f5edc75837272fe91b5c1c1e4a0cea934927cfbc2808dbcf\": rpc error: code = NotFound desc = could not find container \"1ee87bfed553dc07f5edc75837272fe91b5c1c1e4a0cea934927cfbc2808dbcf\": container with ID starting with 1ee87bfed553dc07f5edc75837272fe91b5c1c1e4a0cea934927cfbc2808dbcf not found: ID does not exist" Nov 02 18:12:59.372235 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:59.372213 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-c810ca-5fbdb874f9-2284p"] Nov 02 18:12:59.374592 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:59.374569 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-c810ca-5fbdb874f9-2284p"] Nov 02 18:12:59.412198 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:59.412170 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-c810ca-7c7f98d89-lpltl"] Nov 02 18:12:59.415852 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:59.415830 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-c810ca-7c7f98d89-lpltl"] Nov 02 18:12:59.449008 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:59.448977 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q"] Nov 02 18:12:59.449215 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:59.449190 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q" podUID="411e5db0-b883-4372-b0ce-42ced3dd0a9b" containerName="maas-api" containerID="cri-o://8a337a8562e57563532f886fe2def11cac94cdfbc8649b40b1840ce34cba20b7" gracePeriod=30 Nov 02 18:12:59.738523 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:59.738491 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q" Nov 02 18:12:59.803912 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:59.803877 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/411e5db0-b883-4372-b0ce-42ced3dd0a9b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/411e5db0-b883-4372-b0ce-42ced3dd0a9b-metrics-certs\") pod \"411e5db0-b883-4372-b0ce-42ced3dd0a9b\" (UID: \"411e5db0-b883-4372-b0ce-42ced3dd0a9b\") " Nov 02 18:12:59.804100 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:59.803956 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/411e5db0-b883-4372-b0ce-42ced3dd0a9b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/411e5db0-b883-4372-b0ce-42ced3dd0a9b-openshift-service-ca\") pod \"411e5db0-b883-4372-b0ce-42ced3dd0a9b\" (UID: \"411e5db0-b883-4372-b0ce-42ced3dd0a9b\") " Nov 02 18:12:59.804100 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:59.804073 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/411e5db0-b883-4372-b0ce-42ced3dd0a9b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/411e5db0-b883-4372-b0ce-42ced3dd0a9b-maas-api-tls\") pod \"411e5db0-b883-4372-b0ce-42ced3dd0a9b\" (UID: \"411e5db0-b883-4372-b0ce-42ced3dd0a9b\") " Nov 02 18:12:59.804231 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:59.804173 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/411e5db0-b883-4372-b0ce-42ced3dd0a9b-kube-api-access-xdv4w\" (UniqueName: \"kubernetes.io/projected/411e5db0-b883-4372-b0ce-42ced3dd0a9b-kube-api-access-xdv4w\") pod \"411e5db0-b883-4372-b0ce-42ced3dd0a9b\" (UID: \"411e5db0-b883-4372-b0ce-42ced3dd0a9b\") " Nov 02 18:12:59.804400 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:59.804375 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/411e5db0-b883-4372-b0ce-42ced3dd0a9b-openshift-service-ca" pod "411e5db0-b883-4372-b0ce-42ced3dd0a9b" (UID: "411e5db0-b883-4372-b0ce-42ced3dd0a9b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:59.804689 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:59.804666 4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/411e5db0-b883-4372-b0ce-42ced3dd0a9b-openshift-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:59.806618 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:59.806580 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/411e5db0-b883-4372-b0ce-42ced3dd0a9b-maas-api-tls" pod "411e5db0-b883-4372-b0ce-42ced3dd0a9b" (UID: "411e5db0-b883-4372-b0ce-42ced3dd0a9b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:12:59.806839 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:59.806813 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/411e5db0-b883-4372-b0ce-42ced3dd0a9b-kube-api-access-xdv4w" pod "411e5db0-b883-4372-b0ce-42ced3dd0a9b" (UID: "411e5db0-b883-4372-b0ce-42ced3dd0a9b"). InnerVolumeSpecName "kube-api-access-xdv4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:59.806998 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:59.806974 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/411e5db0-b883-4372-b0ce-42ced3dd0a9b-metrics-certs" pod "411e5db0-b883-4372-b0ce-42ced3dd0a9b" (UID: "411e5db0-b883-4372-b0ce-42ced3dd0a9b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:12:59.919212 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:59.919166 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/411e5db0-b883-4372-b0ce-42ced3dd0a9b-maas-api-tls\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:59.919212 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:59.919212 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdv4w\" (UniqueName: \"kubernetes.io/projected/411e5db0-b883-4372-b0ce-42ced3dd0a9b-kube-api-access-xdv4w\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:12:59.919533 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:12:59.919227 4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/411e5db0-b883-4372-b0ce-42ced3dd0a9b-metrics-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:00.326868 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:00.326819 4471 generic.go:356] "Generic (PLEG): container finished" podID="411e5db0-b883-4372-b0ce-42ced3dd0a9b" containerID="8a337a8562e57563532f886fe2def11cac94cdfbc8649b40b1840ce34cba20b7" exitCode=0 Nov 02 18:13:00.326868 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:00.326873 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q" Nov 02 18:13:00.327465 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:00.326887 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q" event={"ID":"411e5db0-b883-4372-b0ce-42ced3dd0a9b","Type":"ContainerDied","Data":"8a337a8562e57563532f886fe2def11cac94cdfbc8649b40b1840ce34cba20b7"} Nov 02 18:13:00.327465 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:00.326965 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q" event={"ID":"411e5db0-b883-4372-b0ce-42ced3dd0a9b","Type":"ContainerDied","Data":"324c3386958002fc3398d2dab194107d1b3b5c86193a1cfc660457112f8b0c7e"} Nov 02 18:13:00.327465 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:00.326999 4471 scope.go:122] "RemoveContainer" containerID="8a337a8562e57563532f886fe2def11cac94cdfbc8649b40b1840ce34cba20b7" Nov 02 18:13:00.341769 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:00.341745 4471 scope.go:122] "RemoveContainer" containerID="8a337a8562e57563532f886fe2def11cac94cdfbc8649b40b1840ce34cba20b7" Nov 02 18:13:00.342057 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:13:00.342036 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a337a8562e57563532f886fe2def11cac94cdfbc8649b40b1840ce34cba20b7\": container with ID starting with 8a337a8562e57563532f886fe2def11cac94cdfbc8649b40b1840ce34cba20b7 not found: ID does not exist" containerID="8a337a8562e57563532f886fe2def11cac94cdfbc8649b40b1840ce34cba20b7" Nov 02 18:13:00.342119 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:00.342066 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a337a8562e57563532f886fe2def11cac94cdfbc8649b40b1840ce34cba20b7"} err="failed to get container status \"8a337a8562e57563532f886fe2def11cac94cdfbc8649b40b1840ce34cba20b7\": rpc error: code = NotFound desc = could not find container \"8a337a8562e57563532f886fe2def11cac94cdfbc8649b40b1840ce34cba20b7\": container with ID starting with 8a337a8562e57563532f886fe2def11cac94cdfbc8649b40b1840ce34cba20b7 not found: ID does not exist" Nov 02 18:13:00.381566 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:00.381527 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q"] Nov 02 18:13:00.393716 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:00.393679 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-227939-7db554747d-7kz4q"] Nov 02 18:13:00.441764 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:00.441734 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0381ed09-c883-448a-baf2-c7cde6ed90ab" path="/var/lib/kubelet/pods/0381ed09-c883-448a-baf2-c7cde6ed90ab/volumes" Nov 02 18:13:00.442183 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:00.442169 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1408655c-5efa-4883-b9d2-0b51327d40bb" path="/var/lib/kubelet/pods/1408655c-5efa-4883-b9d2-0b51327d40bb/volumes" Nov 02 18:13:00.442630 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:00.442615 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="411e5db0-b883-4372-b0ce-42ced3dd0a9b" path="/var/lib/kubelet/pods/411e5db0-b883-4372-b0ce-42ced3dd0a9b/volumes" Nov 02 18:13:00.443353 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:00.443336 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7395fee0-6fb5-4a90-be40-ba36fde4b163" path="/var/lib/kubelet/pods/7395fee0-6fb5-4a90-be40-ba36fde4b163/volumes" Nov 02 18:13:00.443748 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:00.443734 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3f2ca77-1f97-44a4-99f1-291daab2dee3" path="/var/lib/kubelet/pods/c3f2ca77-1f97-44a4-99f1-291daab2dee3/volumes" Nov 02 18:13:00.922198 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:00.922161 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:13:01.461330 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:01.460427 4471 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:13:01.461330 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:01.460496 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" podUID="c987fa61-24a5-40f2-956f-70f3947bdcd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:13:03.389335 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:03.389288 4471 generic.go:356] "Generic (PLEG): container finished" podID="c987fa61-24a5-40f2-956f-70f3947bdcd7" containerID="6182dc3e28ebfb10afbe89a2fd4b4998635ab1802458412842a544fad88fd2bb" exitCode=0 Nov 02 18:13:03.389685 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:03.389333 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" event={"ID":"c987fa61-24a5-40f2-956f-70f3947bdcd7","Type":"ContainerDied","Data":"6182dc3e28ebfb10afbe89a2fd4b4998635ab1802458412842a544fad88fd2bb"} Nov 02 18:13:03.389728 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:03.389714 4471 scope.go:122] "RemoveContainer" containerID="6182dc3e28ebfb10afbe89a2fd4b4998635ab1802458412842a544fad88fd2bb" Nov 02 18:13:03.391133 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:03.391106 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:13:03.391221 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:03.391172 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:13:03.391221 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:03.391203 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:13:04.400556 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:04.400509 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" event={"ID":"c987fa61-24a5-40f2-956f-70f3947bdcd7","Type":"ContainerStarted","Data":"8b2c6b50148d7d0f73a05b0743164c2e81e5534a63b7a80eeeaea72cc8a7ef8a"} Nov 02 18:13:05.392900 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:05.392868 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 18:13:05.393081 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:05.392961 4471 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 02 18:13:05.393081 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:05.392993 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" podUID="c987fa61-24a5-40f2-956f-70f3947bdcd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 02 18:13:05.619258 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:05.619218 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q"] Nov 02 18:13:05.620022 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:05.619991 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" podUID="fae7257d-c6ad-4e86-8133-aacf8110adb5" containerName="istio-proxy" containerID="cri-o://cae7f6af307bb2b5fad1524414c9317dafb1ac517d675cf661261dffb26772f2" gracePeriod=30 Nov 02 18:13:06.392490 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:06.392452 4471 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 02 18:13:06.392781 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:06.392513 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" podUID="c987fa61-24a5-40f2-956f-70f3947bdcd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 02 18:13:07.183882 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:07.183842 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd"] Nov 02 18:13:07.184383 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:07.184045 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" podUID="3de46b2c-81e7-458e-b693-088684a8a8d8" containerName="istio-proxy" containerID="cri-o://0808e5156af6b05614ccba1c65e8cd336d7aeecf8428100a36b9f35257d806f2" gracePeriod=30 Nov 02 18:13:07.392536 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:07.392495 4471 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 02 18:13:07.392725 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:07.392555 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" podUID="c987fa61-24a5-40f2-956f-70f3947bdcd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 02 18:13:08.392961 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:08.392925 4471 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 02 18:13:08.393388 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:08.392982 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" podUID="c987fa61-24a5-40f2-956f-70f3947bdcd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 02 18:13:09.088389 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:09.088234 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp"] Nov 02 18:13:09.088604 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:09.088470 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" podUID="b54fd98c-e8c7-4578-b027-5275834e8c72" containerName="istio-proxy" containerID="cri-o://71b848ccedd2eb154590ba64e370d24213d32f1bbf81e3d2eee71e5c9e3773f0" gracePeriod=30 Nov 02 18:13:09.237390 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:13:09.237352 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:13:09.392904 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:09.392812 4471 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 02 18:13:09.392904 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:09.392874 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" podUID="c987fa61-24a5-40f2-956f-70f3947bdcd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 02 18:13:09.678634 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:09.678547 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:13:09.678634 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:09.678614 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" podUID="fae7257d-c6ad-4e86-8133-aacf8110adb5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:13:10.396751 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:10.396720 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 18:13:10.397201 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:10.397181 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 18:13:10.397853 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:10.397824 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 18:13:10.537076 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:10.537035 4471 patch_prober.go:29] interesting pod/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:13:10.537248 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:10.537102 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" podUID="b54fd98c-e8c7-4578-b027-5275834e8c72" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:13:10.883430 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:10.883405 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 18:13:11.011213 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.011191 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-workload-socket\") pod \"fae7257d-c6ad-4e86-8133-aacf8110adb5\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " Nov 02 18:13:11.011437 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.011237 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-credential-socket\") pod \"fae7257d-c6ad-4e86-8133-aacf8110adb5\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " Nov 02 18:13:11.011437 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.011278 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-istio-envoy\") pod \"fae7257d-c6ad-4e86-8133-aacf8110adb5\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " Nov 02 18:13:11.011437 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.011327 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-workload-certs\") pod \"fae7257d-c6ad-4e86-8133-aacf8110adb5\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " Nov 02 18:13:11.011437 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.011348 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-istio-data\") pod \"fae7257d-c6ad-4e86-8133-aacf8110adb5\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " Nov 02 18:13:11.011437 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.011404 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fae7257d-c6ad-4e86-8133-aacf8110adb5-kube-api-access-zcxnf\" (UniqueName: \"kubernetes.io/projected/fae7257d-c6ad-4e86-8133-aacf8110adb5-kube-api-access-zcxnf\") pod \"fae7257d-c6ad-4e86-8133-aacf8110adb5\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " Nov 02 18:13:11.011738 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.011503 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fae7257d-c6ad-4e86-8133-aacf8110adb5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fae7257d-c6ad-4e86-8133-aacf8110adb5-istiod-ca-cert\") pod \"fae7257d-c6ad-4e86-8133-aacf8110adb5\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " Nov 02 18:13:11.011738 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.011506 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-workload-socket" pod "fae7257d-c6ad-4e86-8133-aacf8110adb5" (UID: "fae7257d-c6ad-4e86-8133-aacf8110adb5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:11.011738 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.011566 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fae7257d-c6ad-4e86-8133-aacf8110adb5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fae7257d-c6ad-4e86-8133-aacf8110adb5-istio-podinfo\") pod \"fae7257d-c6ad-4e86-8133-aacf8110adb5\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " Nov 02 18:13:11.011738 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.011650 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-workload-certs" pod "fae7257d-c6ad-4e86-8133-aacf8110adb5" (UID: "fae7257d-c6ad-4e86-8133-aacf8110adb5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:11.011738 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.011655 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fae7257d-c6ad-4e86-8133-aacf8110adb5-istio-token\" (UniqueName: \"kubernetes.io/projected/fae7257d-c6ad-4e86-8133-aacf8110adb5-istio-token\") pod \"fae7257d-c6ad-4e86-8133-aacf8110adb5\" (UID: \"fae7257d-c6ad-4e86-8133-aacf8110adb5\") " Nov 02 18:13:11.012002 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.011635 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-credential-socket" pod "fae7257d-c6ad-4e86-8133-aacf8110adb5" (UID: "fae7257d-c6ad-4e86-8133-aacf8110adb5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:11.012002 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.011787 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-istio-data" pod "fae7257d-c6ad-4e86-8133-aacf8110adb5" (UID: "fae7257d-c6ad-4e86-8133-aacf8110adb5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:11.012002 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.011895 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fae7257d-c6ad-4e86-8133-aacf8110adb5-istiod-ca-cert" pod "fae7257d-c6ad-4e86-8133-aacf8110adb5" (UID: "fae7257d-c6ad-4e86-8133-aacf8110adb5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:13:11.012242 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.012220 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:11.012330 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.012250 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:11.012330 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.012271 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:11.012330 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.012287 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:11.012330 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.012300 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fae7257d-c6ad-4e86-8133-aacf8110adb5-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:11.013509 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.013487 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-istio-envoy" pod "fae7257d-c6ad-4e86-8133-aacf8110adb5" (UID: "fae7257d-c6ad-4e86-8133-aacf8110adb5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:11.013769 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.013744 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae7257d-c6ad-4e86-8133-aacf8110adb5-istio-token" pod "fae7257d-c6ad-4e86-8133-aacf8110adb5" (UID: "fae7257d-c6ad-4e86-8133-aacf8110adb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:11.014076 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.014055 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae7257d-c6ad-4e86-8133-aacf8110adb5-kube-api-access-zcxnf" pod "fae7257d-c6ad-4e86-8133-aacf8110adb5" (UID: "fae7257d-c6ad-4e86-8133-aacf8110adb5"). InnerVolumeSpecName "kube-api-access-zcxnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:11.014076 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.014063 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fae7257d-c6ad-4e86-8133-aacf8110adb5-istio-podinfo" pod "fae7257d-c6ad-4e86-8133-aacf8110adb5" (UID: "fae7257d-c6ad-4e86-8133-aacf8110adb5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:13:11.113547 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.113513 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fae7257d-c6ad-4e86-8133-aacf8110adb5-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:11.113547 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.113543 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcxnf\" (UniqueName: \"kubernetes.io/projected/fae7257d-c6ad-4e86-8133-aacf8110adb5-kube-api-access-zcxnf\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:11.113547 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.113553 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fae7257d-c6ad-4e86-8133-aacf8110adb5-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:11.113790 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.113561 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fae7257d-c6ad-4e86-8133-aacf8110adb5-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:11.444688 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.444579 4471 generic.go:356] "Generic (PLEG): container finished" podID="fae7257d-c6ad-4e86-8133-aacf8110adb5" containerID="cae7f6af307bb2b5fad1524414c9317dafb1ac517d675cf661261dffb26772f2" exitCode=0 Nov 02 18:13:11.445147 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.444691 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" Nov 02 18:13:11.445147 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.444769 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" event={"ID":"fae7257d-c6ad-4e86-8133-aacf8110adb5","Type":"ContainerDied","Data":"cae7f6af307bb2b5fad1524414c9317dafb1ac517d675cf661261dffb26772f2"} Nov 02 18:13:11.445147 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.444836 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q" event={"ID":"fae7257d-c6ad-4e86-8133-aacf8110adb5","Type":"ContainerDied","Data":"8192ecc877d1c61c98a83af1e31c123b0aba9847ef9e8f3a9d579c22ed8e23f1"} Nov 02 18:13:11.445147 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.444863 4471 scope.go:122] "RemoveContainer" containerID="cae7f6af307bb2b5fad1524414c9317dafb1ac517d675cf661261dffb26772f2" Nov 02 18:13:11.455884 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.455863 4471 scope.go:122] "RemoveContainer" containerID="cae7f6af307bb2b5fad1524414c9317dafb1ac517d675cf661261dffb26772f2" Nov 02 18:13:11.456153 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:13:11.456131 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cae7f6af307bb2b5fad1524414c9317dafb1ac517d675cf661261dffb26772f2\": container with ID starting with cae7f6af307bb2b5fad1524414c9317dafb1ac517d675cf661261dffb26772f2 not found: ID does not exist" containerID="cae7f6af307bb2b5fad1524414c9317dafb1ac517d675cf661261dffb26772f2" Nov 02 18:13:11.456201 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.456162 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cae7f6af307bb2b5fad1524414c9317dafb1ac517d675cf661261dffb26772f2"} err="failed to get container status \"cae7f6af307bb2b5fad1524414c9317dafb1ac517d675cf661261dffb26772f2\": rpc error: code = NotFound desc = could not find container \"cae7f6af307bb2b5fad1524414c9317dafb1ac517d675cf661261dffb26772f2\": container with ID starting with cae7f6af307bb2b5fad1524414c9317dafb1ac517d675cf661261dffb26772f2 not found: ID does not exist" Nov 02 18:13:11.503574 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.503545 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q"] Nov 02 18:13:11.515830 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.515801 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-jbp6q"] Nov 02 18:13:11.710210 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:11.710129 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q"] Nov 02 18:13:12.439664 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.439632 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 18:13:12.445499 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.443821 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fae7257d-c6ad-4e86-8133-aacf8110adb5" path="/var/lib/kubelet/pods/fae7257d-c6ad-4e86-8133-aacf8110adb5/volumes" Nov 02 18:13:12.455575 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.455508 4471 generic.go:356] "Generic (PLEG): container finished" podID="3de46b2c-81e7-458e-b693-088684a8a8d8" containerID="0808e5156af6b05614ccba1c65e8cd336d7aeecf8428100a36b9f35257d806f2" exitCode=0 Nov 02 18:13:12.455723 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.455591 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" event={"ID":"3de46b2c-81e7-458e-b693-088684a8a8d8","Type":"ContainerDied","Data":"0808e5156af6b05614ccba1c65e8cd336d7aeecf8428100a36b9f35257d806f2"} Nov 02 18:13:12.455723 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.455615 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" Nov 02 18:13:12.455723 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.455648 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd" event={"ID":"3de46b2c-81e7-458e-b693-088684a8a8d8","Type":"ContainerDied","Data":"ca4e75cfd89534938f21011cf065b7abe4b4b8ac071a798887fe06d133913beb"} Nov 02 18:13:12.455723 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.455675 4471 scope.go:122] "RemoveContainer" containerID="0808e5156af6b05614ccba1c65e8cd336d7aeecf8428100a36b9f35257d806f2" Nov 02 18:13:12.457141 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.457116 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" podUID="c987fa61-24a5-40f2-956f-70f3947bdcd7" containerName="istio-proxy" containerID="cri-o://8b2c6b50148d7d0f73a05b0743164c2e81e5534a63b7a80eeeaea72cc8a7ef8a" gracePeriod=30 Nov 02 18:13:12.465696 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.465676 4471 scope.go:122] "RemoveContainer" containerID="8399a59c4ce7fc6b3a252d81d472281b29c65b0724f4fe40d60c4ee606590140" Nov 02 18:13:12.481826 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.481799 4471 scope.go:122] "RemoveContainer" containerID="0808e5156af6b05614ccba1c65e8cd336d7aeecf8428100a36b9f35257d806f2" Nov 02 18:13:12.482126 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:13:12.482100 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0808e5156af6b05614ccba1c65e8cd336d7aeecf8428100a36b9f35257d806f2\": container with ID starting with 0808e5156af6b05614ccba1c65e8cd336d7aeecf8428100a36b9f35257d806f2 not found: ID does not exist" containerID="0808e5156af6b05614ccba1c65e8cd336d7aeecf8428100a36b9f35257d806f2" Nov 02 18:13:12.482220 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.482133 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0808e5156af6b05614ccba1c65e8cd336d7aeecf8428100a36b9f35257d806f2"} err="failed to get container status \"0808e5156af6b05614ccba1c65e8cd336d7aeecf8428100a36b9f35257d806f2\": rpc error: code = NotFound desc = could not find container \"0808e5156af6b05614ccba1c65e8cd336d7aeecf8428100a36b9f35257d806f2\": container with ID starting with 0808e5156af6b05614ccba1c65e8cd336d7aeecf8428100a36b9f35257d806f2 not found: ID does not exist" Nov 02 18:13:12.482220 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.482154 4471 scope.go:122] "RemoveContainer" containerID="8399a59c4ce7fc6b3a252d81d472281b29c65b0724f4fe40d60c4ee606590140" Nov 02 18:13:12.482452 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:13:12.482436 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8399a59c4ce7fc6b3a252d81d472281b29c65b0724f4fe40d60c4ee606590140\": container with ID starting with 8399a59c4ce7fc6b3a252d81d472281b29c65b0724f4fe40d60c4ee606590140 not found: ID does not exist" containerID="8399a59c4ce7fc6b3a252d81d472281b29c65b0724f4fe40d60c4ee606590140" Nov 02 18:13:12.482502 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.482458 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8399a59c4ce7fc6b3a252d81d472281b29c65b0724f4fe40d60c4ee606590140"} err="failed to get container status \"8399a59c4ce7fc6b3a252d81d472281b29c65b0724f4fe40d60c4ee606590140\": rpc error: code = NotFound desc = could not find container \"8399a59c4ce7fc6b3a252d81d472281b29c65b0724f4fe40d60c4ee606590140\": container with ID starting with 8399a59c4ce7fc6b3a252d81d472281b29c65b0724f4fe40d60c4ee606590140 not found: ID does not exist" Nov 02 18:13:12.529528 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.529497 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-workload-certs\") pod \"3de46b2c-81e7-458e-b693-088684a8a8d8\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " Nov 02 18:13:12.529724 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.529569 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-istio-data\") pod \"3de46b2c-81e7-458e-b693-088684a8a8d8\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " Nov 02 18:13:12.529724 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.529602 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-credential-socket\") pod \"3de46b2c-81e7-458e-b693-088684a8a8d8\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " Nov 02 18:13:12.529724 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.529628 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-workload-socket\") pod \"3de46b2c-81e7-458e-b693-088684a8a8d8\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " Nov 02 18:13:12.529724 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.529689 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3de46b2c-81e7-458e-b693-088684a8a8d8-kube-api-access-rfc98\" (UniqueName: \"kubernetes.io/projected/3de46b2c-81e7-458e-b693-088684a8a8d8-kube-api-access-rfc98\") pod \"3de46b2c-81e7-458e-b693-088684a8a8d8\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " Nov 02 18:13:12.529951 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.529755 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-istio-envoy\") pod \"3de46b2c-81e7-458e-b693-088684a8a8d8\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " Nov 02 18:13:12.529951 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.529810 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3de46b2c-81e7-458e-b693-088684a8a8d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de46b2c-81e7-458e-b693-088684a8a8d8-istiod-ca-cert\") pod \"3de46b2c-81e7-458e-b693-088684a8a8d8\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " Nov 02 18:13:12.529951 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.529831 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-istio-data" pod "3de46b2c-81e7-458e-b693-088684a8a8d8" (UID: "3de46b2c-81e7-458e-b693-088684a8a8d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:12.529951 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.529855 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3de46b2c-81e7-458e-b693-088684a8a8d8-istio-token\" (UniqueName: \"kubernetes.io/projected/3de46b2c-81e7-458e-b693-088684a8a8d8-istio-token\") pod \"3de46b2c-81e7-458e-b693-088684a8a8d8\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " Nov 02 18:13:12.529951 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.529856 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-workload-certs" pod "3de46b2c-81e7-458e-b693-088684a8a8d8" (UID: "3de46b2c-81e7-458e-b693-088684a8a8d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:12.529951 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.529924 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3de46b2c-81e7-458e-b693-088684a8a8d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de46b2c-81e7-458e-b693-088684a8a8d8-istio-podinfo\") pod \"3de46b2c-81e7-458e-b693-088684a8a8d8\" (UID: \"3de46b2c-81e7-458e-b693-088684a8a8d8\") " Nov 02 18:13:12.530290 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.529928 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-credential-socket" pod "3de46b2c-81e7-458e-b693-088684a8a8d8" (UID: "3de46b2c-81e7-458e-b693-088684a8a8d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:12.530290 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.530008 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-workload-socket" pod "3de46b2c-81e7-458e-b693-088684a8a8d8" (UID: "3de46b2c-81e7-458e-b693-088684a8a8d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:12.530290 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.530173 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3de46b2c-81e7-458e-b693-088684a8a8d8-istiod-ca-cert" pod "3de46b2c-81e7-458e-b693-088684a8a8d8" (UID: "3de46b2c-81e7-458e-b693-088684a8a8d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:13:12.530845 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.530819 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:12.530951 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.530849 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:12.530951 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.530865 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:12.530951 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.530881 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:12.530951 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.530897 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de46b2c-81e7-458e-b693-088684a8a8d8-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:12.531953 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.531929 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-istio-envoy" pod "3de46b2c-81e7-458e-b693-088684a8a8d8" (UID: "3de46b2c-81e7-458e-b693-088684a8a8d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:12.532089 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.532073 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3de46b2c-81e7-458e-b693-088684a8a8d8-istio-podinfo" pod "3de46b2c-81e7-458e-b693-088684a8a8d8" (UID: "3de46b2c-81e7-458e-b693-088684a8a8d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:13:12.532161 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.532084 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de46b2c-81e7-458e-b693-088684a8a8d8-istio-token" pod "3de46b2c-81e7-458e-b693-088684a8a8d8" (UID: "3de46b2c-81e7-458e-b693-088684a8a8d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:12.532161 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.532115 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de46b2c-81e7-458e-b693-088684a8a8d8-kube-api-access-rfc98" pod "3de46b2c-81e7-458e-b693-088684a8a8d8" (UID: "3de46b2c-81e7-458e-b693-088684a8a8d8"). InnerVolumeSpecName "kube-api-access-rfc98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:12.603825 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.603797 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j"] Nov 02 18:13:12.631917 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.631877 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfc98\" (UniqueName: \"kubernetes.io/projected/3de46b2c-81e7-458e-b693-088684a8a8d8-kube-api-access-rfc98\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:12.631917 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.631898 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2"] Nov 02 18:13:12.632161 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.631909 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de46b2c-81e7-458e-b693-088684a8a8d8-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:12.632161 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.632000 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3de46b2c-81e7-458e-b693-088684a8a8d8-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:12.632161 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.632010 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de46b2c-81e7-458e-b693-088684a8a8d8-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:12.815682 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.815647 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd"] Nov 02 18:13:12.819230 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:12.819192 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f6da5f-openshift-default-8d788b577-pg6xd"] Nov 02 18:13:14.346169 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.346143 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:13:14.441261 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.441174 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3de46b2c-81e7-458e-b693-088684a8a8d8" path="/var/lib/kubelet/pods/3de46b2c-81e7-458e-b693-088684a8a8d8/volumes" Nov 02 18:13:14.463607 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.463579 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b54fd98c-e8c7-4578-b027-5275834e8c72-istio-token\" (UniqueName: \"kubernetes.io/projected/b54fd98c-e8c7-4578-b027-5275834e8c72-istio-token\") pod \"b54fd98c-e8c7-4578-b027-5275834e8c72\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " Nov 02 18:13:14.463773 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.463638 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b54fd98c-e8c7-4578-b027-5275834e8c72-kube-api-access-5hv6r\" (UniqueName: \"kubernetes.io/projected/b54fd98c-e8c7-4578-b027-5275834e8c72-kube-api-access-5hv6r\") pod \"b54fd98c-e8c7-4578-b027-5275834e8c72\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " Nov 02 18:13:14.463773 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.463702 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-istio-envoy\") pod \"b54fd98c-e8c7-4578-b027-5275834e8c72\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " Nov 02 18:13:14.463773 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.463721 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b54fd98c-e8c7-4578-b027-5275834e8c72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b54fd98c-e8c7-4578-b027-5275834e8c72-istio-podinfo\") pod \"b54fd98c-e8c7-4578-b027-5275834e8c72\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " Nov 02 18:13:14.463773 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.463744 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-workload-socket\") pod \"b54fd98c-e8c7-4578-b027-5275834e8c72\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " Nov 02 18:13:14.464001 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.463862 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b54fd98c-e8c7-4578-b027-5275834e8c72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b54fd98c-e8c7-4578-b027-5275834e8c72-istiod-ca-cert\") pod \"b54fd98c-e8c7-4578-b027-5275834e8c72\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " Nov 02 18:13:14.464001 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.463913 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-istio-data\") pod \"b54fd98c-e8c7-4578-b027-5275834e8c72\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " Nov 02 18:13:14.464001 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.463963 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-credential-socket\") pod \"b54fd98c-e8c7-4578-b027-5275834e8c72\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " Nov 02 18:13:14.464001 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.463996 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-workload-certs\") pod \"b54fd98c-e8c7-4578-b027-5275834e8c72\" (UID: \"b54fd98c-e8c7-4578-b027-5275834e8c72\") " Nov 02 18:13:14.464225 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.464041 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-workload-socket" pod "b54fd98c-e8c7-4578-b027-5275834e8c72" (UID: "b54fd98c-e8c7-4578-b027-5275834e8c72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:14.464295 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.464278 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-istio-data" pod "b54fd98c-e8c7-4578-b027-5275834e8c72" (UID: "b54fd98c-e8c7-4578-b027-5275834e8c72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:14.464404 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.464299 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-credential-socket" pod "b54fd98c-e8c7-4578-b027-5275834e8c72" (UID: "b54fd98c-e8c7-4578-b027-5275834e8c72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:14.464582 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.464511 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-workload-certs" pod "b54fd98c-e8c7-4578-b027-5275834e8c72" (UID: "b54fd98c-e8c7-4578-b027-5275834e8c72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:14.464719 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.464608 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b54fd98c-e8c7-4578-b027-5275834e8c72-istiod-ca-cert" pod "b54fd98c-e8c7-4578-b027-5275834e8c72" (UID: "b54fd98c-e8c7-4578-b027-5275834e8c72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:13:14.464824 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.464790 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:14.464824 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.464816 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b54fd98c-e8c7-4578-b027-5275834e8c72-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:14.464953 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.464832 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:14.464953 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.464845 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:14.464953 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.464861 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:14.466066 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.466040 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b54fd98c-e8c7-4578-b027-5275834e8c72-istio-token" pod "b54fd98c-e8c7-4578-b027-5275834e8c72" (UID: "b54fd98c-e8c7-4578-b027-5275834e8c72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:14.466165 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.466068 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b54fd98c-e8c7-4578-b027-5275834e8c72-kube-api-access-5hv6r" pod "b54fd98c-e8c7-4578-b027-5275834e8c72" (UID: "b54fd98c-e8c7-4578-b027-5275834e8c72"). InnerVolumeSpecName "kube-api-access-5hv6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:14.466165 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.466102 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-istio-envoy" pod "b54fd98c-e8c7-4578-b027-5275834e8c72" (UID: "b54fd98c-e8c7-4578-b027-5275834e8c72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:14.466362 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.466341 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b54fd98c-e8c7-4578-b027-5275834e8c72-istio-podinfo" pod "b54fd98c-e8c7-4578-b027-5275834e8c72" (UID: "b54fd98c-e8c7-4578-b027-5275834e8c72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:13:14.471850 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.471827 4471 generic.go:356] "Generic (PLEG): container finished" podID="b54fd98c-e8c7-4578-b027-5275834e8c72" containerID="71b848ccedd2eb154590ba64e370d24213d32f1bbf81e3d2eee71e5c9e3773f0" exitCode=0 Nov 02 18:13:14.471963 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.471875 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" Nov 02 18:13:14.471963 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.471906 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" event={"ID":"b54fd98c-e8c7-4578-b027-5275834e8c72","Type":"ContainerDied","Data":"71b848ccedd2eb154590ba64e370d24213d32f1bbf81e3d2eee71e5c9e3773f0"} Nov 02 18:13:14.471963 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.471961 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp" event={"ID":"b54fd98c-e8c7-4578-b027-5275834e8c72","Type":"ContainerDied","Data":"26d2a8115d26cb1769844f4ef656622e339c98b99f5d4669f7af578d70929da0"} Nov 02 18:13:14.472090 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.471991 4471 scope.go:122] "RemoveContainer" containerID="71b848ccedd2eb154590ba64e370d24213d32f1bbf81e3d2eee71e5c9e3773f0" Nov 02 18:13:14.494715 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.494696 4471 scope.go:122] "RemoveContainer" containerID="e515c5200f7cf6ac9ec59af8a997c85c4400ba615caac97d7a3d496945f2f677" Nov 02 18:13:14.505722 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.505699 4471 scope.go:122] "RemoveContainer" containerID="71b848ccedd2eb154590ba64e370d24213d32f1bbf81e3d2eee71e5c9e3773f0" Nov 02 18:13:14.505998 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:13:14.505978 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71b848ccedd2eb154590ba64e370d24213d32f1bbf81e3d2eee71e5c9e3773f0\": container with ID starting with 71b848ccedd2eb154590ba64e370d24213d32f1bbf81e3d2eee71e5c9e3773f0 not found: ID does not exist" containerID="71b848ccedd2eb154590ba64e370d24213d32f1bbf81e3d2eee71e5c9e3773f0" Nov 02 18:13:14.506048 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.506012 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71b848ccedd2eb154590ba64e370d24213d32f1bbf81e3d2eee71e5c9e3773f0"} err="failed to get container status \"71b848ccedd2eb154590ba64e370d24213d32f1bbf81e3d2eee71e5c9e3773f0\": rpc error: code = NotFound desc = could not find container \"71b848ccedd2eb154590ba64e370d24213d32f1bbf81e3d2eee71e5c9e3773f0\": container with ID starting with 71b848ccedd2eb154590ba64e370d24213d32f1bbf81e3d2eee71e5c9e3773f0 not found: ID does not exist" Nov 02 18:13:14.506048 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.506033 4471 scope.go:122] "RemoveContainer" containerID="e515c5200f7cf6ac9ec59af8a997c85c4400ba615caac97d7a3d496945f2f677" Nov 02 18:13:14.506256 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:13:14.506237 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e515c5200f7cf6ac9ec59af8a997c85c4400ba615caac97d7a3d496945f2f677\": container with ID starting with e515c5200f7cf6ac9ec59af8a997c85c4400ba615caac97d7a3d496945f2f677 not found: ID does not exist" containerID="e515c5200f7cf6ac9ec59af8a997c85c4400ba615caac97d7a3d496945f2f677" Nov 02 18:13:14.506400 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.506256 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e515c5200f7cf6ac9ec59af8a997c85c4400ba615caac97d7a3d496945f2f677"} err="failed to get container status \"e515c5200f7cf6ac9ec59af8a997c85c4400ba615caac97d7a3d496945f2f677\": rpc error: code = NotFound desc = could not find container \"e515c5200f7cf6ac9ec59af8a997c85c4400ba615caac97d7a3d496945f2f677\": container with ID starting with e515c5200f7cf6ac9ec59af8a997c85c4400ba615caac97d7a3d496945f2f677 not found: ID does not exist" Nov 02 18:13:14.537741 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.537706 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp"] Nov 02 18:13:14.549710 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.549682 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-227939-openshift-default-86b8b557d7-2tfwp"] Nov 02 18:13:14.566121 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.566099 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b54fd98c-e8c7-4578-b027-5275834e8c72-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:14.566121 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.566120 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hv6r\" (UniqueName: \"kubernetes.io/projected/b54fd98c-e8c7-4578-b027-5275834e8c72-kube-api-access-5hv6r\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:14.566268 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.566132 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b54fd98c-e8c7-4578-b027-5275834e8c72-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:14.566268 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:14.566143 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b54fd98c-e8c7-4578-b027-5275834e8c72-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:16.440463 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:16.440430 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b54fd98c-e8c7-4578-b027-5275834e8c72" path="/var/lib/kubelet/pods/b54fd98c-e8c7-4578-b027-5275834e8c72/volumes" Nov 02 18:13:17.713510 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:17.713483 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 18:13:17.800168 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:17.800133 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-istio-envoy\") pod \"c987fa61-24a5-40f2-956f-70f3947bdcd7\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " Nov 02 18:13:17.800411 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:17.800205 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c987fa61-24a5-40f2-956f-70f3947bdcd7-kube-api-access-8vq4k\" (UniqueName: \"kubernetes.io/projected/c987fa61-24a5-40f2-956f-70f3947bdcd7-kube-api-access-8vq4k\") pod \"c987fa61-24a5-40f2-956f-70f3947bdcd7\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " Nov 02 18:13:17.800411 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:17.800252 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-credential-socket\") pod \"c987fa61-24a5-40f2-956f-70f3947bdcd7\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " Nov 02 18:13:17.800578 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:17.800410 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c987fa61-24a5-40f2-956f-70f3947bdcd7-istio-token\" (UniqueName: \"kubernetes.io/projected/c987fa61-24a5-40f2-956f-70f3947bdcd7-istio-token\") pod \"c987fa61-24a5-40f2-956f-70f3947bdcd7\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " Nov 02 18:13:17.800578 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:17.800465 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-workload-socket\") pod \"c987fa61-24a5-40f2-956f-70f3947bdcd7\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " Nov 02 18:13:17.800578 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:17.800498 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-workload-certs\") pod \"c987fa61-24a5-40f2-956f-70f3947bdcd7\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " Nov 02 18:13:17.800578 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:17.800531 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-istio-data\") pod \"c987fa61-24a5-40f2-956f-70f3947bdcd7\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " Nov 02 18:13:17.800811 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:17.800580 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-credential-socket" pod "c987fa61-24a5-40f2-956f-70f3947bdcd7" (UID: "c987fa61-24a5-40f2-956f-70f3947bdcd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:17.800811 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:17.800642 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c987fa61-24a5-40f2-956f-70f3947bdcd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c987fa61-24a5-40f2-956f-70f3947bdcd7-istio-podinfo\") pod \"c987fa61-24a5-40f2-956f-70f3947bdcd7\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " Nov 02 18:13:17.800811 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:17.800700 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c987fa61-24a5-40f2-956f-70f3947bdcd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c987fa61-24a5-40f2-956f-70f3947bdcd7-istiod-ca-cert\") pod \"c987fa61-24a5-40f2-956f-70f3947bdcd7\" (UID: \"c987fa61-24a5-40f2-956f-70f3947bdcd7\") " Nov 02 18:13:17.800811 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:17.800786 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-workload-socket" pod "c987fa61-24a5-40f2-956f-70f3947bdcd7" (UID: "c987fa61-24a5-40f2-956f-70f3947bdcd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:17.801046 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:17.800985 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-istio-data" pod "c987fa61-24a5-40f2-956f-70f3947bdcd7" (UID: "c987fa61-24a5-40f2-956f-70f3947bdcd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:17.801340 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:17.801235 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:17.801340 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:17.801262 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:17.801340 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:17.801280 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:17.801340 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:17.801240 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-workload-certs" pod "c987fa61-24a5-40f2-956f-70f3947bdcd7" (UID: "c987fa61-24a5-40f2-956f-70f3947bdcd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:17.801651 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:17.801347 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c987fa61-24a5-40f2-956f-70f3947bdcd7-istiod-ca-cert" pod "c987fa61-24a5-40f2-956f-70f3947bdcd7" (UID: "c987fa61-24a5-40f2-956f-70f3947bdcd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:13:17.802649 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:17.802622 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-istio-envoy" pod "c987fa61-24a5-40f2-956f-70f3947bdcd7" (UID: "c987fa61-24a5-40f2-956f-70f3947bdcd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:17.803162 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:17.803141 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c987fa61-24a5-40f2-956f-70f3947bdcd7-istio-token" pod "c987fa61-24a5-40f2-956f-70f3947bdcd7" (UID: "c987fa61-24a5-40f2-956f-70f3947bdcd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:17.803268 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:17.803253 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c987fa61-24a5-40f2-956f-70f3947bdcd7-kube-api-access-8vq4k" pod "c987fa61-24a5-40f2-956f-70f3947bdcd7" (UID: "c987fa61-24a5-40f2-956f-70f3947bdcd7"). InnerVolumeSpecName "kube-api-access-8vq4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:17.803412 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:17.803390 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c987fa61-24a5-40f2-956f-70f3947bdcd7-istio-podinfo" pod "c987fa61-24a5-40f2-956f-70f3947bdcd7" (UID: "c987fa61-24a5-40f2-956f-70f3947bdcd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:13:17.902836 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:17.902801 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c987fa61-24a5-40f2-956f-70f3947bdcd7-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:17.902836 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:17.902831 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c987fa61-24a5-40f2-956f-70f3947bdcd7-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:17.902836 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:17.902840 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:17.903087 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:17.902849 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vq4k\" (UniqueName: \"kubernetes.io/projected/c987fa61-24a5-40f2-956f-70f3947bdcd7-kube-api-access-8vq4k\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:17.903087 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:17.902883 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c987fa61-24a5-40f2-956f-70f3947bdcd7-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:17.903087 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:17.902892 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c987fa61-24a5-40f2-956f-70f3947bdcd7-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:18.501581 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:18.501543 4471 generic.go:356] "Generic (PLEG): container finished" podID="c987fa61-24a5-40f2-956f-70f3947bdcd7" containerID="8b2c6b50148d7d0f73a05b0743164c2e81e5534a63b7a80eeeaea72cc8a7ef8a" exitCode=0 Nov 02 18:13:18.501744 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:18.501612 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" Nov 02 18:13:18.501744 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:18.501625 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" event={"ID":"c987fa61-24a5-40f2-956f-70f3947bdcd7","Type":"ContainerDied","Data":"8b2c6b50148d7d0f73a05b0743164c2e81e5534a63b7a80eeeaea72cc8a7ef8a"} Nov 02 18:13:18.501744 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:18.501687 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q" event={"ID":"c987fa61-24a5-40f2-956f-70f3947bdcd7","Type":"ContainerDied","Data":"2a7ea84d137b91c8e68418cc5911f02452253a00e13cef1e872755fb74fe3e5e"} Nov 02 18:13:18.501744 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:18.501709 4471 scope.go:122] "RemoveContainer" containerID="8b2c6b50148d7d0f73a05b0743164c2e81e5534a63b7a80eeeaea72cc8a7ef8a" Nov 02 18:13:18.511809 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:18.511784 4471 scope.go:122] "RemoveContainer" containerID="6182dc3e28ebfb10afbe89a2fd4b4998635ab1802458412842a544fad88fd2bb" Nov 02 18:13:18.521078 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:18.521057 4471 scope.go:122] "RemoveContainer" containerID="8b2c6b50148d7d0f73a05b0743164c2e81e5534a63b7a80eeeaea72cc8a7ef8a" Nov 02 18:13:18.521407 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:13:18.521385 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b2c6b50148d7d0f73a05b0743164c2e81e5534a63b7a80eeeaea72cc8a7ef8a\": container with ID starting with 8b2c6b50148d7d0f73a05b0743164c2e81e5534a63b7a80eeeaea72cc8a7ef8a not found: ID does not exist" containerID="8b2c6b50148d7d0f73a05b0743164c2e81e5534a63b7a80eeeaea72cc8a7ef8a" Nov 02 18:13:18.521462 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:18.521415 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b2c6b50148d7d0f73a05b0743164c2e81e5534a63b7a80eeeaea72cc8a7ef8a"} err="failed to get container status \"8b2c6b50148d7d0f73a05b0743164c2e81e5534a63b7a80eeeaea72cc8a7ef8a\": rpc error: code = NotFound desc = could not find container \"8b2c6b50148d7d0f73a05b0743164c2e81e5534a63b7a80eeeaea72cc8a7ef8a\": container with ID starting with 8b2c6b50148d7d0f73a05b0743164c2e81e5534a63b7a80eeeaea72cc8a7ef8a not found: ID does not exist" Nov 02 18:13:18.521462 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:18.521433 4471 scope.go:122] "RemoveContainer" containerID="6182dc3e28ebfb10afbe89a2fd4b4998635ab1802458412842a544fad88fd2bb" Nov 02 18:13:18.521718 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:13:18.521698 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6182dc3e28ebfb10afbe89a2fd4b4998635ab1802458412842a544fad88fd2bb\": container with ID starting with 6182dc3e28ebfb10afbe89a2fd4b4998635ab1802458412842a544fad88fd2bb not found: ID does not exist" containerID="6182dc3e28ebfb10afbe89a2fd4b4998635ab1802458412842a544fad88fd2bb" Nov 02 18:13:18.521788 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:18.521723 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6182dc3e28ebfb10afbe89a2fd4b4998635ab1802458412842a544fad88fd2bb"} err="failed to get container status \"6182dc3e28ebfb10afbe89a2fd4b4998635ab1802458412842a544fad88fd2bb\": rpc error: code = NotFound desc = could not find container \"6182dc3e28ebfb10afbe89a2fd4b4998635ab1802458412842a544fad88fd2bb\": container with ID starting with 6182dc3e28ebfb10afbe89a2fd4b4998635ab1802458412842a544fad88fd2bb not found: ID does not exist" Nov 02 18:13:18.559273 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:18.559236 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q"] Nov 02 18:13:18.571895 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:18.571861 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2672ed-openshift-default-678d94ccf7-zxr8q"] Nov 02 18:13:19.331795 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:19.331762 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d"] Nov 02 18:13:20.441240 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:20.441205 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c987fa61-24a5-40f2-956f-70f3947bdcd7" path="/var/lib/kubelet/pods/c987fa61-24a5-40f2-956f-70f3947bdcd7/volumes" Nov 02 18:13:27.624230 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:27.624193 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" containerID="cri-o://2306a65cf2a35c75f09a07854c5c8050020be427883c4c8dc5943663abe6ad84" gracePeriod=45 Nov 02 18:13:27.651585 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:27.651554 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" containerID="cri-o://40366d0d43189fab94fb9070497b79fc4a1bceaf15279f58aaaff3220b485f13" gracePeriod=45 Nov 02 18:13:27.917288 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:27.917259 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 18:13:27.937088 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:27.937063 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 18:13:28.042190 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.042151 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1c66dee-ht7cn"] Nov 02 18:13:28.044758 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.044729 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1c66dee-ht7cn"] Nov 02 18:13:28.104360 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.104297 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/537a8e6f-da32-4e42-a7a1-bd7c061eabea-dshm\" (UniqueName: \"kubernetes.io/empty-dir/537a8e6f-da32-4e42-a7a1-bd7c061eabea-dshm\") pod \"537a8e6f-da32-4e42-a7a1-bd7c061eabea\" (UID: \"537a8e6f-da32-4e42-a7a1-bd7c061eabea\") " Nov 02 18:13:28.104539 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.104378 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-tmp-dir\") pod \"3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5\" (UID: \"3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5\") " Nov 02 18:13:28.104539 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.104421 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-tls-certs\") pod \"3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5\" (UID: \"3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5\") " Nov 02 18:13:28.104539 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.104471 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/537a8e6f-da32-4e42-a7a1-bd7c061eabea-tls-certs\" (UniqueName: \"kubernetes.io/secret/537a8e6f-da32-4e42-a7a1-bd7c061eabea-tls-certs\") pod \"537a8e6f-da32-4e42-a7a1-bd7c061eabea\" (UID: \"537a8e6f-da32-4e42-a7a1-bd7c061eabea\") " Nov 02 18:13:28.104539 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.104508 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/537a8e6f-da32-4e42-a7a1-bd7c061eabea-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/537a8e6f-da32-4e42-a7a1-bd7c061eabea-model-cache\") pod \"537a8e6f-da32-4e42-a7a1-bd7c061eabea\" (UID: \"537a8e6f-da32-4e42-a7a1-bd7c061eabea\") " Nov 02 18:13:28.104776 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.104597 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-kube-api-access-rjpxh\" (UniqueName: \"kubernetes.io/projected/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-kube-api-access-rjpxh\") pod \"3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5\" (UID: \"3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5\") " Nov 02 18:13:28.104776 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.104627 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/537a8e6f-da32-4e42-a7a1-bd7c061eabea-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/537a8e6f-da32-4e42-a7a1-bd7c061eabea-tmp-dir\") pod \"537a8e6f-da32-4e42-a7a1-bd7c061eabea\" (UID: \"537a8e6f-da32-4e42-a7a1-bd7c061eabea\") " Nov 02 18:13:28.104776 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.104673 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-home\" (UniqueName: \"kubernetes.io/empty-dir/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-home\") pod \"3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5\" (UID: \"3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5\") " Nov 02 18:13:28.104776 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.104682 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-tmp-dir" pod "3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" (UID: "3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:28.105606 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.104893 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-home" pod "3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" (UID: "3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:28.105606 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.104896 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-model-cache\") pod \"3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5\" (UID: \"3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5\") " Nov 02 18:13:28.105606 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.104974 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/537a8e6f-da32-4e42-a7a1-bd7c061eabea-kube-api-access-nvwft\" (UniqueName: \"kubernetes.io/projected/537a8e6f-da32-4e42-a7a1-bd7c061eabea-kube-api-access-nvwft\") pod \"537a8e6f-da32-4e42-a7a1-bd7c061eabea\" (UID: \"537a8e6f-da32-4e42-a7a1-bd7c061eabea\") " Nov 02 18:13:28.105606 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.105006 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/537a8e6f-da32-4e42-a7a1-bd7c061eabea-home\" (UniqueName: \"kubernetes.io/empty-dir/537a8e6f-da32-4e42-a7a1-bd7c061eabea-home\") pod \"537a8e6f-da32-4e42-a7a1-bd7c061eabea\" (UID: \"537a8e6f-da32-4e42-a7a1-bd7c061eabea\") " Nov 02 18:13:28.105606 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.105067 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-model-cache" pod "3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" (UID: "3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:28.105606 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.105073 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-dshm\") pod \"3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5\" (UID: \"3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5\") " Nov 02 18:13:28.105606 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.105295 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/537a8e6f-da32-4e42-a7a1-bd7c061eabea-home" pod "537a8e6f-da32-4e42-a7a1-bd7c061eabea" (UID: "537a8e6f-da32-4e42-a7a1-bd7c061eabea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:28.105606 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.105587 4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-tmp-dir\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:28.105606 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.105609 4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-home\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:28.106112 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.105622 4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-model-cache\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:28.106112 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.105637 4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/537a8e6f-da32-4e42-a7a1-bd7c061eabea-home\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:28.106112 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.105634 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/537a8e6f-da32-4e42-a7a1-bd7c061eabea-model-cache" pod "537a8e6f-da32-4e42-a7a1-bd7c061eabea" (UID: "537a8e6f-da32-4e42-a7a1-bd7c061eabea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:28.106112 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.105701 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/537a8e6f-da32-4e42-a7a1-bd7c061eabea-tmp-dir" pod "537a8e6f-da32-4e42-a7a1-bd7c061eabea" (UID: "537a8e6f-da32-4e42-a7a1-bd7c061eabea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:28.107094 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.107064 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/537a8e6f-da32-4e42-a7a1-bd7c061eabea-dshm" pod "537a8e6f-da32-4e42-a7a1-bd7c061eabea" (UID: "537a8e6f-da32-4e42-a7a1-bd7c061eabea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:28.107094 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.107089 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-tls-certs" pod "3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" (UID: "3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:13:28.107253 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.107156 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-kube-api-access-rjpxh" pod "3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" (UID: "3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5"). InnerVolumeSpecName "kube-api-access-rjpxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:28.107624 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.107603 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-dshm" pod "3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" (UID: "3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:28.107723 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.107694 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/537a8e6f-da32-4e42-a7a1-bd7c061eabea-tls-certs" pod "537a8e6f-da32-4e42-a7a1-bd7c061eabea" (UID: "537a8e6f-da32-4e42-a7a1-bd7c061eabea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:13:28.107811 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.107793 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/537a8e6f-da32-4e42-a7a1-bd7c061eabea-kube-api-access-nvwft" pod "537a8e6f-da32-4e42-a7a1-bd7c061eabea" (UID: "537a8e6f-da32-4e42-a7a1-bd7c061eabea"). InnerVolumeSpecName "kube-api-access-nvwft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:28.206653 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.206572 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvwft\" (UniqueName: \"kubernetes.io/projected/537a8e6f-da32-4e42-a7a1-bd7c061eabea-kube-api-access-nvwft\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:28.206653 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.206602 4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-dshm\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:28.206653 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.206612 4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/537a8e6f-da32-4e42-a7a1-bd7c061eabea-dshm\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:28.206653 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.206620 4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-tls-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:28.206653 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.206630 4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/537a8e6f-da32-4e42-a7a1-bd7c061eabea-tls-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:28.206653 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.206638 4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/537a8e6f-da32-4e42-a7a1-bd7c061eabea-model-cache\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:28.206653 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.206660 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjpxh\" (UniqueName: \"kubernetes.io/projected/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5-kube-api-access-rjpxh\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:28.207007 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.206668 4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/537a8e6f-da32-4e42-a7a1-bd7c061eabea-tmp-dir\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:28.439760 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.439729 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9f3b98a-f972-4127-a97f-d8c1d1937394" path="/var/lib/kubelet/pods/e9f3b98a-f972-4127-a97f-d8c1d1937394/volumes" Nov 02 18:13:28.577615 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.577577 4471 generic.go:356] "Generic (PLEG): container finished" podID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerID="40366d0d43189fab94fb9070497b79fc4a1bceaf15279f58aaaff3220b485f13" exitCode=0 Nov 02 18:13:28.577796 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.577626 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" Nov 02 18:13:28.577875 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.577825 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" event={"ID":"537a8e6f-da32-4e42-a7a1-bd7c061eabea","Type":"ContainerDied","Data":"40366d0d43189fab94fb9070497b79fc4a1bceaf15279f58aaaff3220b485f13"} Nov 02 18:13:28.577875 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.577864 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2" event={"ID":"537a8e6f-da32-4e42-a7a1-bd7c061eabea","Type":"ContainerDied","Data":"42e8332285812657ae7deb0d48bbf3dc4ba63f6d9f06d315356b7b6fb7a8ef5a"} Nov 02 18:13:28.577951 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.577887 4471 scope.go:122] "RemoveContainer" containerID="40366d0d43189fab94fb9070497b79fc4a1bceaf15279f58aaaff3220b485f13" Nov 02 18:13:28.580300 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.580265 4471 generic.go:356] "Generic (PLEG): container finished" podID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerID="2306a65cf2a35c75f09a07854c5c8050020be427883c4c8dc5943663abe6ad84" exitCode=0 Nov 02 18:13:28.580477 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.580351 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" Nov 02 18:13:28.580477 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.580370 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" event={"ID":"3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5","Type":"ContainerDied","Data":"2306a65cf2a35c75f09a07854c5c8050020be427883c4c8dc5943663abe6ad84"} Nov 02 18:13:28.580477 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.580403 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j" event={"ID":"3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5","Type":"ContainerDied","Data":"ff26e7fc28dde8e4228695dfcf4490c9df228f9b373fce4a9c70f96ff098bd94"} Nov 02 18:13:28.587978 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.587959 4471 scope.go:122] "RemoveContainer" containerID="40366d0d43189fab94fb9070497b79fc4a1bceaf15279f58aaaff3220b485f13" Nov 02 18:13:28.588244 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:13:28.588225 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40366d0d43189fab94fb9070497b79fc4a1bceaf15279f58aaaff3220b485f13\": container with ID starting with 40366d0d43189fab94fb9070497b79fc4a1bceaf15279f58aaaff3220b485f13 not found: ID does not exist" containerID="40366d0d43189fab94fb9070497b79fc4a1bceaf15279f58aaaff3220b485f13" Nov 02 18:13:28.588298 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.588252 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40366d0d43189fab94fb9070497b79fc4a1bceaf15279f58aaaff3220b485f13"} err="failed to get container status \"40366d0d43189fab94fb9070497b79fc4a1bceaf15279f58aaaff3220b485f13\": rpc error: code = NotFound desc = could not find container \"40366d0d43189fab94fb9070497b79fc4a1bceaf15279f58aaaff3220b485f13\": container with ID starting with 40366d0d43189fab94fb9070497b79fc4a1bceaf15279f58aaaff3220b485f13 not found: ID does not exist" Nov 02 18:13:28.588298 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.588269 4471 scope.go:122] "RemoveContainer" containerID="2306a65cf2a35c75f09a07854c5c8050020be427883c4c8dc5943663abe6ad84" Nov 02 18:13:28.596696 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.596678 4471 scope.go:122] "RemoveContainer" containerID="2306a65cf2a35c75f09a07854c5c8050020be427883c4c8dc5943663abe6ad84" Nov 02 18:13:28.596946 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:13:28.596924 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2306a65cf2a35c75f09a07854c5c8050020be427883c4c8dc5943663abe6ad84\": container with ID starting with 2306a65cf2a35c75f09a07854c5c8050020be427883c4c8dc5943663abe6ad84 not found: ID does not exist" containerID="2306a65cf2a35c75f09a07854c5c8050020be427883c4c8dc5943663abe6ad84" Nov 02 18:13:28.597023 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.596960 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2306a65cf2a35c75f09a07854c5c8050020be427883c4c8dc5943663abe6ad84"} err="failed to get container status \"2306a65cf2a35c75f09a07854c5c8050020be427883c4c8dc5943663abe6ad84\": rpc error: code = NotFound desc = could not find container \"2306a65cf2a35c75f09a07854c5c8050020be427883c4c8dc5943663abe6ad84\": container with ID starting with 2306a65cf2a35c75f09a07854c5c8050020be427883c4c8dc5943663abe6ad84 not found: ID does not exist" Nov 02 18:13:28.631386 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.631356 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2"] Nov 02 18:13:28.644420 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.644393 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-7df526/premium-w6-7df526-kserve-57bd79fd6b-6vnm2"] Nov 02 18:13:28.669177 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.669153 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j"] Nov 02 18:13:28.672949 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:28.672927 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-7df526/distinct-2-w6-7df526-kserve-85bc59dfcd-bb76j"] Nov 02 18:13:30.440475 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:30.440442 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" path="/var/lib/kubelet/pods/3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5/volumes" Nov 02 18:13:30.440891 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:30.440876 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" path="/var/lib/kubelet/pods/537a8e6f-da32-4e42-a7a1-bd7c061eabea/volumes" Nov 02 18:13:31.911421 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:31.911385 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6"] Nov 02 18:13:31.911852 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:31.911626 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6" podUID="4d5d1db2-0240-442b-aaed-b2931e9e8d6a" containerName="maas-api" containerID="cri-o://f70838619a28f2c43aaf4898798aa5898183aa45ec70deeacfce753b3c981a3e" gracePeriod=30 Nov 02 18:13:32.192483 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:32.192455 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6" Nov 02 18:13:32.262340 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:32.262288 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d5d1db2-0240-442b-aaed-b2931e9e8d6a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d5d1db2-0240-442b-aaed-b2931e9e8d6a-openshift-service-ca\") pod \"4d5d1db2-0240-442b-aaed-b2931e9e8d6a\" (UID: \"4d5d1db2-0240-442b-aaed-b2931e9e8d6a\") " Nov 02 18:13:32.262528 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:32.262416 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d5d1db2-0240-442b-aaed-b2931e9e8d6a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d5d1db2-0240-442b-aaed-b2931e9e8d6a-metrics-certs\") pod \"4d5d1db2-0240-442b-aaed-b2931e9e8d6a\" (UID: \"4d5d1db2-0240-442b-aaed-b2931e9e8d6a\") " Nov 02 18:13:32.262528 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:32.262449 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d5d1db2-0240-442b-aaed-b2931e9e8d6a-kube-api-access-fbn5r\" (UniqueName: \"kubernetes.io/projected/4d5d1db2-0240-442b-aaed-b2931e9e8d6a-kube-api-access-fbn5r\") pod \"4d5d1db2-0240-442b-aaed-b2931e9e8d6a\" (UID: \"4d5d1db2-0240-442b-aaed-b2931e9e8d6a\") " Nov 02 18:13:32.262528 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:32.262489 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d5d1db2-0240-442b-aaed-b2931e9e8d6a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d5d1db2-0240-442b-aaed-b2931e9e8d6a-maas-api-tls\") pod \"4d5d1db2-0240-442b-aaed-b2931e9e8d6a\" (UID: \"4d5d1db2-0240-442b-aaed-b2931e9e8d6a\") " Nov 02 18:13:32.262709 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:32.262687 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d5d1db2-0240-442b-aaed-b2931e9e8d6a-openshift-service-ca" pod "4d5d1db2-0240-442b-aaed-b2931e9e8d6a" (UID: "4d5d1db2-0240-442b-aaed-b2931e9e8d6a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:13:32.262992 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:32.262956 4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d5d1db2-0240-442b-aaed-b2931e9e8d6a-openshift-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:32.265744 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:32.265714 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d5d1db2-0240-442b-aaed-b2931e9e8d6a-kube-api-access-fbn5r" pod "4d5d1db2-0240-442b-aaed-b2931e9e8d6a" (UID: "4d5d1db2-0240-442b-aaed-b2931e9e8d6a"). InnerVolumeSpecName "kube-api-access-fbn5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:32.265744 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:32.265726 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d5d1db2-0240-442b-aaed-b2931e9e8d6a-metrics-certs" pod "4d5d1db2-0240-442b-aaed-b2931e9e8d6a" (UID: "4d5d1db2-0240-442b-aaed-b2931e9e8d6a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:13:32.265905 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:32.265781 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d5d1db2-0240-442b-aaed-b2931e9e8d6a-maas-api-tls" pod "4d5d1db2-0240-442b-aaed-b2931e9e8d6a" (UID: "4d5d1db2-0240-442b-aaed-b2931e9e8d6a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:13:32.367405 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:32.365972 4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d5d1db2-0240-442b-aaed-b2931e9e8d6a-metrics-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:32.367405 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:32.366015 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbn5r\" (UniqueName: \"kubernetes.io/projected/4d5d1db2-0240-442b-aaed-b2931e9e8d6a-kube-api-access-fbn5r\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:32.367405 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:32.366031 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d5d1db2-0240-442b-aaed-b2931e9e8d6a-maas-api-tls\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:32.606800 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:32.606762 4471 generic.go:356] "Generic (PLEG): container finished" podID="4d5d1db2-0240-442b-aaed-b2931e9e8d6a" containerID="f70838619a28f2c43aaf4898798aa5898183aa45ec70deeacfce753b3c981a3e" exitCode=0 Nov 02 18:13:32.606956 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:32.606827 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6" Nov 02 18:13:32.606956 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:32.606838 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6" event={"ID":"4d5d1db2-0240-442b-aaed-b2931e9e8d6a","Type":"ContainerDied","Data":"f70838619a28f2c43aaf4898798aa5898183aa45ec70deeacfce753b3c981a3e"} Nov 02 18:13:32.606956 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:32.606890 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6" event={"ID":"4d5d1db2-0240-442b-aaed-b2931e9e8d6a","Type":"ContainerDied","Data":"bcb2bded9562805957ac41a7a94f18761dd9cf2a8505b0ebbe83a3f4a70065c0"} Nov 02 18:13:32.606956 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:32.606919 4471 scope.go:122] "RemoveContainer" containerID="f70838619a28f2c43aaf4898798aa5898183aa45ec70deeacfce753b3c981a3e" Nov 02 18:13:32.617801 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:32.617766 4471 scope.go:122] "RemoveContainer" containerID="f70838619a28f2c43aaf4898798aa5898183aa45ec70deeacfce753b3c981a3e" Nov 02 18:13:32.618110 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:13:32.618085 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f70838619a28f2c43aaf4898798aa5898183aa45ec70deeacfce753b3c981a3e\": container with ID starting with f70838619a28f2c43aaf4898798aa5898183aa45ec70deeacfce753b3c981a3e not found: ID does not exist" containerID="f70838619a28f2c43aaf4898798aa5898183aa45ec70deeacfce753b3c981a3e" Nov 02 18:13:32.618204 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:32.618117 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f70838619a28f2c43aaf4898798aa5898183aa45ec70deeacfce753b3c981a3e"} err="failed to get container status \"f70838619a28f2c43aaf4898798aa5898183aa45ec70deeacfce753b3c981a3e\": rpc error: code = NotFound desc = could not find container \"f70838619a28f2c43aaf4898798aa5898183aa45ec70deeacfce753b3c981a3e\": container with ID starting with f70838619a28f2c43aaf4898798aa5898183aa45ec70deeacfce753b3c981a3e not found: ID does not exist" Nov 02 18:13:32.664141 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:32.664104 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6"] Nov 02 18:13:32.677092 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:32.677054 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-7df526-6f77fb8569-h2gk6"] Nov 02 18:13:34.314108 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.314069 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb"] Nov 02 18:13:34.314614 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.314364 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb" podUID="c5bf24bf-3188-425c-bced-4e77490d6377" containerName="maas-api" containerID="cri-o://6e333e4b1477287274e52483305b3ca94097b6e76232be4049723c5ea6c421fc" gracePeriod=30 Nov 02 18:13:34.352763 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.352595 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" containerID="cri-o://56ae181916890cc38e68e75e804f52118d0ca1f6441c80a08fff91b0552615ed" gracePeriod=45 Nov 02 18:13:34.439619 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.439588 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d5d1db2-0240-442b-aaed-b2931e9e8d6a" path="/var/lib/kubelet/pods/4d5d1db2-0240-442b-aaed-b2931e9e8d6a/volumes" Nov 02 18:13:34.622091 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.622059 4471 generic.go:356] "Generic (PLEG): container finished" podID="c5bf24bf-3188-425c-bced-4e77490d6377" containerID="6e333e4b1477287274e52483305b3ca94097b6e76232be4049723c5ea6c421fc" exitCode=0 Nov 02 18:13:34.622235 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.622125 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb" event={"ID":"c5bf24bf-3188-425c-bced-4e77490d6377","Type":"ContainerDied","Data":"6e333e4b1477287274e52483305b3ca94097b6e76232be4049723c5ea6c421fc"} Nov 02 18:13:34.622235 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.622181 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb" event={"ID":"c5bf24bf-3188-425c-bced-4e77490d6377","Type":"ContainerDied","Data":"f9b618f5202ec85f7b167761ac5655d4e8e1ae4a2d7f92e18c4c427d86128290"} Nov 02 18:13:34.622235 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.622207 4471 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9b618f5202ec85f7b167761ac5655d4e8e1ae4a2d7f92e18c4c427d86128290" Nov 02 18:13:34.622422 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.622401 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb" Nov 02 18:13:34.623862 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.623842 4471 generic.go:356] "Generic (PLEG): container finished" podID="9e729ed8-8407-477c-90a7-c899203f2452" containerID="56ae181916890cc38e68e75e804f52118d0ca1f6441c80a08fff91b0552615ed" exitCode=0 Nov 02 18:13:34.623980 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.623915 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" event={"ID":"9e729ed8-8407-477c-90a7-c899203f2452","Type":"ContainerDied","Data":"56ae181916890cc38e68e75e804f52118d0ca1f6441c80a08fff91b0552615ed"} Nov 02 18:13:34.639726 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.639706 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 18:13:34.704681 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.704647 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e729ed8-8407-477c-90a7-c899203f2452-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e729ed8-8407-477c-90a7-c899203f2452-tls-certs\") pod \"9e729ed8-8407-477c-90a7-c899203f2452\" (UID: \"9e729ed8-8407-477c-90a7-c899203f2452\") " Nov 02 18:13:34.704873 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.704695 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e729ed8-8407-477c-90a7-c899203f2452-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e729ed8-8407-477c-90a7-c899203f2452-tmp-dir\") pod \"9e729ed8-8407-477c-90a7-c899203f2452\" (UID: \"9e729ed8-8407-477c-90a7-c899203f2452\") " Nov 02 18:13:34.704873 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.704736 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e729ed8-8407-477c-90a7-c899203f2452-kube-api-access-svtt8\" (UniqueName: \"kubernetes.io/projected/9e729ed8-8407-477c-90a7-c899203f2452-kube-api-access-svtt8\") pod \"9e729ed8-8407-477c-90a7-c899203f2452\" (UID: \"9e729ed8-8407-477c-90a7-c899203f2452\") " Nov 02 18:13:34.704873 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.704770 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e729ed8-8407-477c-90a7-c899203f2452-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e729ed8-8407-477c-90a7-c899203f2452-model-cache\") pod \"9e729ed8-8407-477c-90a7-c899203f2452\" (UID: \"9e729ed8-8407-477c-90a7-c899203f2452\") " Nov 02 18:13:34.704873 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.704835 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e729ed8-8407-477c-90a7-c899203f2452-home\" (UniqueName: \"kubernetes.io/empty-dir/9e729ed8-8407-477c-90a7-c899203f2452-home\") pod \"9e729ed8-8407-477c-90a7-c899203f2452\" (UID: \"9e729ed8-8407-477c-90a7-c899203f2452\") " Nov 02 18:13:34.705101 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.704896 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e729ed8-8407-477c-90a7-c899203f2452-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e729ed8-8407-477c-90a7-c899203f2452-dshm\") pod \"9e729ed8-8407-477c-90a7-c899203f2452\" (UID: \"9e729ed8-8407-477c-90a7-c899203f2452\") " Nov 02 18:13:34.705101 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.704965 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e729ed8-8407-477c-90a7-c899203f2452-tmp-dir" pod "9e729ed8-8407-477c-90a7-c899203f2452" (UID: "9e729ed8-8407-477c-90a7-c899203f2452"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:34.705101 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.704986 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5bf24bf-3188-425c-bced-4e77490d6377-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5bf24bf-3188-425c-bced-4e77490d6377-openshift-service-ca\") pod \"c5bf24bf-3188-425c-bced-4e77490d6377\" (UID: \"c5bf24bf-3188-425c-bced-4e77490d6377\") " Nov 02 18:13:34.705101 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.705030 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5bf24bf-3188-425c-bced-4e77490d6377-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5bf24bf-3188-425c-bced-4e77490d6377-maas-api-tls\") pod \"c5bf24bf-3188-425c-bced-4e77490d6377\" (UID: \"c5bf24bf-3188-425c-bced-4e77490d6377\") " Nov 02 18:13:34.705101 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.705070 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5bf24bf-3188-425c-bced-4e77490d6377-kube-api-access-bmhtt\" (UniqueName: \"kubernetes.io/projected/c5bf24bf-3188-425c-bced-4e77490d6377-kube-api-access-bmhtt\") pod \"c5bf24bf-3188-425c-bced-4e77490d6377\" (UID: \"c5bf24bf-3188-425c-bced-4e77490d6377\") " Nov 02 18:13:34.705101 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.705092 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e729ed8-8407-477c-90a7-c899203f2452-model-cache" pod "9e729ed8-8407-477c-90a7-c899203f2452" (UID: "9e729ed8-8407-477c-90a7-c899203f2452"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:34.706721 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.705112 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5bf24bf-3188-425c-bced-4e77490d6377-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5bf24bf-3188-425c-bced-4e77490d6377-metrics-certs\") pod \"c5bf24bf-3188-425c-bced-4e77490d6377\" (UID: \"c5bf24bf-3188-425c-bced-4e77490d6377\") " Nov 02 18:13:34.706721 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.705708 4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e729ed8-8407-477c-90a7-c899203f2452-tmp-dir\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:34.706721 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.705732 4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e729ed8-8407-477c-90a7-c899203f2452-model-cache\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:34.706721 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.706473 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e729ed8-8407-477c-90a7-c899203f2452-home" pod "9e729ed8-8407-477c-90a7-c899203f2452" (UID: "9e729ed8-8407-477c-90a7-c899203f2452"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:34.706721 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.706680 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5bf24bf-3188-425c-bced-4e77490d6377-openshift-service-ca" pod "c5bf24bf-3188-425c-bced-4e77490d6377" (UID: "c5bf24bf-3188-425c-bced-4e77490d6377"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:13:34.708339 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.708157 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5bf24bf-3188-425c-bced-4e77490d6377-maas-api-tls" pod "c5bf24bf-3188-425c-bced-4e77490d6377" (UID: "c5bf24bf-3188-425c-bced-4e77490d6377"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:13:34.708339 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.708266 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e729ed8-8407-477c-90a7-c899203f2452-tls-certs" pod "9e729ed8-8407-477c-90a7-c899203f2452" (UID: "9e729ed8-8407-477c-90a7-c899203f2452"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:13:34.709053 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.709026 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5bf24bf-3188-425c-bced-4e77490d6377-kube-api-access-bmhtt" pod "c5bf24bf-3188-425c-bced-4e77490d6377" (UID: "c5bf24bf-3188-425c-bced-4e77490d6377"). InnerVolumeSpecName "kube-api-access-bmhtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:34.709053 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.709037 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5bf24bf-3188-425c-bced-4e77490d6377-metrics-certs" pod "c5bf24bf-3188-425c-bced-4e77490d6377" (UID: "c5bf24bf-3188-425c-bced-4e77490d6377"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:13:34.711428 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.711396 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e729ed8-8407-477c-90a7-c899203f2452-dshm" pod "9e729ed8-8407-477c-90a7-c899203f2452" (UID: "9e729ed8-8407-477c-90a7-c899203f2452"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:34.714576 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.714550 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e729ed8-8407-477c-90a7-c899203f2452-kube-api-access-svtt8" pod "9e729ed8-8407-477c-90a7-c899203f2452" (UID: "9e729ed8-8407-477c-90a7-c899203f2452"). InnerVolumeSpecName "kube-api-access-svtt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:34.806969 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.806929 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmhtt\" (UniqueName: \"kubernetes.io/projected/c5bf24bf-3188-425c-bced-4e77490d6377-kube-api-access-bmhtt\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:34.807194 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.806975 4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5bf24bf-3188-425c-bced-4e77490d6377-metrics-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:34.807194 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.806992 4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e729ed8-8407-477c-90a7-c899203f2452-tls-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:34.807194 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.807006 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svtt8\" (UniqueName: \"kubernetes.io/projected/9e729ed8-8407-477c-90a7-c899203f2452-kube-api-access-svtt8\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:34.807194 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.807024 4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e729ed8-8407-477c-90a7-c899203f2452-home\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:34.807194 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.807036 4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e729ed8-8407-477c-90a7-c899203f2452-dshm\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:34.807194 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.807050 4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5bf24bf-3188-425c-bced-4e77490d6377-openshift-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:34.807194 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:34.807067 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5bf24bf-3188-425c-bced-4e77490d6377-maas-api-tls\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:35.630320 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:35.630233 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" Nov 02 18:13:35.630320 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:35.630263 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d" event={"ID":"9e729ed8-8407-477c-90a7-c899203f2452","Type":"ContainerDied","Data":"19940231b65540a954ceb42998284bbf43949a37eee22860434fe737cee80857"} Nov 02 18:13:35.630788 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:35.630326 4471 scope.go:122] "RemoveContainer" containerID="56ae181916890cc38e68e75e804f52118d0ca1f6441c80a08fff91b0552615ed" Nov 02 18:13:35.630788 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:35.630457 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb" Nov 02 18:13:35.684270 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:35.684242 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb"] Nov 02 18:13:35.696688 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:35.696664 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5eedab-7f5b47cfbb-l7dmb"] Nov 02 18:13:35.727141 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:35.727118 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d"] Nov 02 18:13:35.731260 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:35.731240 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-863b87/premium-w5-863b87-kserve-cfb8948c5-t2x8d"] Nov 02 18:13:36.439495 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:36.439463 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e729ed8-8407-477c-90a7-c899203f2452" path="/var/lib/kubelet/pods/9e729ed8-8407-477c-90a7-c899203f2452/volumes" Nov 02 18:13:36.439877 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:36.439862 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5bf24bf-3188-425c-bced-4e77490d6377" path="/var/lib/kubelet/pods/c5bf24bf-3188-425c-bced-4e77490d6377/volumes" Nov 02 18:13:39.183237 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:13:39.183196 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:13:39.712471 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:39.712435 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z"] Nov 02 18:13:39.712751 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:39.712718 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" podUID="db524df0-282c-43eb-936b-53ed7d9388cf" containerName="istio-proxy" containerID="cri-o://233bdc170834e90343633263b943122af0ad834932a7aa1adccf6a682acc8239" gracePeriod=30 Nov 02 18:13:41.508481 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:41.508445 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:13:44.032348 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:44.032298 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a161d95e-bmz7c"] Nov 02 18:13:44.044860 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:44.044834 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a161d95e-bmz7c"] Nov 02 18:13:44.439630 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:44.439555 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5278da20-920d-4ed6-8d05-f477d8d628bf" path="/var/lib/kubelet/pods/5278da20-920d-4ed6-8d05-f477d8d628bf/volumes" Nov 02 18:13:44.959176 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:44.959143 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 18:13:45.050644 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.050609 4471 scope.go:122] "RemoveContainer" containerID="4ca6c89eda89cedd6c28328c6c5825b82ec5d26142af68e6b0e01f39d21f6002" Nov 02 18:13:45.060256 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.060232 4471 scope.go:122] "RemoveContainer" containerID="233bdc170834e90343633263b943122af0ad834932a7aa1adccf6a682acc8239" Nov 02 18:13:45.069027 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.069005 4471 scope.go:122] "RemoveContainer" containerID="6e333e4b1477287274e52483305b3ca94097b6e76232be4049723c5ea6c421fc" Nov 02 18:13:45.111358 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.111328 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-workload-certs\") pod \"db524df0-282c-43eb-936b-53ed7d9388cf\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " Nov 02 18:13:45.111448 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.111375 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-credential-socket\") pod \"db524df0-282c-43eb-936b-53ed7d9388cf\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " Nov 02 18:13:45.111448 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.111409 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db524df0-282c-43eb-936b-53ed7d9388cf-kube-api-access-2zlvt\" (UniqueName: \"kubernetes.io/projected/db524df0-282c-43eb-936b-53ed7d9388cf-kube-api-access-2zlvt\") pod \"db524df0-282c-43eb-936b-53ed7d9388cf\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " Nov 02 18:13:45.111448 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.111438 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db524df0-282c-43eb-936b-53ed7d9388cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db524df0-282c-43eb-936b-53ed7d9388cf-istio-podinfo\") pod \"db524df0-282c-43eb-936b-53ed7d9388cf\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " Nov 02 18:13:45.111616 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.111464 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db524df0-282c-43eb-936b-53ed7d9388cf-istio-token\" (UniqueName: \"kubernetes.io/projected/db524df0-282c-43eb-936b-53ed7d9388cf-istio-token\") pod \"db524df0-282c-43eb-936b-53ed7d9388cf\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " Nov 02 18:13:45.111616 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.111505 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db524df0-282c-43eb-936b-53ed7d9388cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db524df0-282c-43eb-936b-53ed7d9388cf-istiod-ca-cert\") pod \"db524df0-282c-43eb-936b-53ed7d9388cf\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " Nov 02 18:13:45.111616 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.111535 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-workload-socket\") pod \"db524df0-282c-43eb-936b-53ed7d9388cf\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " Nov 02 18:13:45.111616 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.111600 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-istio-envoy\") pod \"db524df0-282c-43eb-936b-53ed7d9388cf\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " Nov 02 18:13:45.111892 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.111639 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-istio-data\") pod \"db524df0-282c-43eb-936b-53ed7d9388cf\" (UID: \"db524df0-282c-43eb-936b-53ed7d9388cf\") " Nov 02 18:13:45.111892 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.111642 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-workload-certs" pod "db524df0-282c-43eb-936b-53ed7d9388cf" (UID: "db524df0-282c-43eb-936b-53ed7d9388cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:45.111892 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.111666 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-credential-socket" pod "db524df0-282c-43eb-936b-53ed7d9388cf" (UID: "db524df0-282c-43eb-936b-53ed7d9388cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:45.111892 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.111882 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-workload-socket" pod "db524df0-282c-43eb-936b-53ed7d9388cf" (UID: "db524df0-282c-43eb-936b-53ed7d9388cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:45.112121 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.111912 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-istio-data" pod "db524df0-282c-43eb-936b-53ed7d9388cf" (UID: "db524df0-282c-43eb-936b-53ed7d9388cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:45.112217 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.112196 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:45.112279 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.112228 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:45.112279 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.112243 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:45.112279 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.112256 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:45.112279 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.112200 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db524df0-282c-43eb-936b-53ed7d9388cf-istiod-ca-cert" pod "db524df0-282c-43eb-936b-53ed7d9388cf" (UID: "db524df0-282c-43eb-936b-53ed7d9388cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:13:45.113838 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.113816 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db524df0-282c-43eb-936b-53ed7d9388cf-kube-api-access-2zlvt" pod "db524df0-282c-43eb-936b-53ed7d9388cf" (UID: "db524df0-282c-43eb-936b-53ed7d9388cf"). InnerVolumeSpecName "kube-api-access-2zlvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:45.113940 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.113846 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db524df0-282c-43eb-936b-53ed7d9388cf-istio-token" pod "db524df0-282c-43eb-936b-53ed7d9388cf" (UID: "db524df0-282c-43eb-936b-53ed7d9388cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:45.114065 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.114038 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db524df0-282c-43eb-936b-53ed7d9388cf-istio-podinfo" pod "db524df0-282c-43eb-936b-53ed7d9388cf" (UID: "db524df0-282c-43eb-936b-53ed7d9388cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:13:45.114159 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.114098 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-istio-envoy" pod "db524df0-282c-43eb-936b-53ed7d9388cf" (UID: "db524df0-282c-43eb-936b-53ed7d9388cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:45.212989 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.212959 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zlvt\" (UniqueName: \"kubernetes.io/projected/db524df0-282c-43eb-936b-53ed7d9388cf-kube-api-access-2zlvt\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:45.212989 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.212988 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db524df0-282c-43eb-936b-53ed7d9388cf-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:45.212989 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.212997 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db524df0-282c-43eb-936b-53ed7d9388cf-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:45.213265 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.213005 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db524df0-282c-43eb-936b-53ed7d9388cf-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:45.213265 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.213012 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db524df0-282c-43eb-936b-53ed7d9388cf-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:45.705371 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.705338 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" Nov 02 18:13:45.705541 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.705335 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" event={"ID":"db524df0-282c-43eb-936b-53ed7d9388cf","Type":"ContainerDied","Data":"233bdc170834e90343633263b943122af0ad834932a7aa1adccf6a682acc8239"} Nov 02 18:13:45.705541 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.705475 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z" event={"ID":"db524df0-282c-43eb-936b-53ed7d9388cf","Type":"ContainerDied","Data":"97d50e56be368bfa139ffc947eb40e86229fb8f4d2f5191af670f8d52cc09775"} Nov 02 18:13:45.850237 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.850207 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z"] Nov 02 18:13:45.855224 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:45.855194 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-lxm4z"] Nov 02 18:13:46.440526 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:46.440490 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db524df0-282c-43eb-936b-53ed7d9388cf" path="/var/lib/kubelet/pods/db524df0-282c-43eb-936b-53ed7d9388cf/volumes" Nov 02 18:13:51.417691 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:51.417658 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-2899ab-59796778-drqbh"] Nov 02 18:13:51.418144 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:51.417886 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2899ab-59796778-drqbh" podUID="3f849213-a9f5-4236-ba98-a41e70d67dc5" containerName="maas-api" containerID="cri-o://8bfb140074f3f645a0eb964263edf59bd150a3f532d6006c7ed43f1c15254cbb" gracePeriod=30 Nov 02 18:13:51.662778 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:51.662753 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2899ab-59796778-drqbh" Nov 02 18:13:51.737172 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:51.737082 4471 generic.go:356] "Generic (PLEG): container finished" podID="3f849213-a9f5-4236-ba98-a41e70d67dc5" containerID="8bfb140074f3f645a0eb964263edf59bd150a3f532d6006c7ed43f1c15254cbb" exitCode=0 Nov 02 18:13:51.737172 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:51.737130 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2899ab-59796778-drqbh" Nov 02 18:13:51.737413 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:51.737172 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2899ab-59796778-drqbh" event={"ID":"3f849213-a9f5-4236-ba98-a41e70d67dc5","Type":"ContainerDied","Data":"8bfb140074f3f645a0eb964263edf59bd150a3f532d6006c7ed43f1c15254cbb"} Nov 02 18:13:51.737413 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:51.737247 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2899ab-59796778-drqbh" event={"ID":"3f849213-a9f5-4236-ba98-a41e70d67dc5","Type":"ContainerDied","Data":"99b917bdc4405764774567b2a03e216b02ca118e3cfbd715adda0f1dc5bddc31"} Nov 02 18:13:51.737413 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:51.737272 4471 scope.go:122] "RemoveContainer" containerID="8bfb140074f3f645a0eb964263edf59bd150a3f532d6006c7ed43f1c15254cbb" Nov 02 18:13:51.746068 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:51.746049 4471 scope.go:122] "RemoveContainer" containerID="8bfb140074f3f645a0eb964263edf59bd150a3f532d6006c7ed43f1c15254cbb" Nov 02 18:13:51.746358 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:13:51.746340 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bfb140074f3f645a0eb964263edf59bd150a3f532d6006c7ed43f1c15254cbb\": container with ID starting with 8bfb140074f3f645a0eb964263edf59bd150a3f532d6006c7ed43f1c15254cbb not found: ID does not exist" containerID="8bfb140074f3f645a0eb964263edf59bd150a3f532d6006c7ed43f1c15254cbb" Nov 02 18:13:51.746429 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:51.746374 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bfb140074f3f645a0eb964263edf59bd150a3f532d6006c7ed43f1c15254cbb"} err="failed to get container status \"8bfb140074f3f645a0eb964263edf59bd150a3f532d6006c7ed43f1c15254cbb\": rpc error: code = NotFound desc = could not find container \"8bfb140074f3f645a0eb964263edf59bd150a3f532d6006c7ed43f1c15254cbb\": container with ID starting with 8bfb140074f3f645a0eb964263edf59bd150a3f532d6006c7ed43f1c15254cbb not found: ID does not exist" Nov 02 18:13:51.784658 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:51.784636 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f849213-a9f5-4236-ba98-a41e70d67dc5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f849213-a9f5-4236-ba98-a41e70d67dc5-metrics-certs\") pod \"3f849213-a9f5-4236-ba98-a41e70d67dc5\" (UID: \"3f849213-a9f5-4236-ba98-a41e70d67dc5\") " Nov 02 18:13:51.784785 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:51.784678 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f849213-a9f5-4236-ba98-a41e70d67dc5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f849213-a9f5-4236-ba98-a41e70d67dc5-maas-api-tls\") pod \"3f849213-a9f5-4236-ba98-a41e70d67dc5\" (UID: \"3f849213-a9f5-4236-ba98-a41e70d67dc5\") " Nov 02 18:13:51.784785 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:51.784728 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f849213-a9f5-4236-ba98-a41e70d67dc5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f849213-a9f5-4236-ba98-a41e70d67dc5-openshift-service-ca\") pod \"3f849213-a9f5-4236-ba98-a41e70d67dc5\" (UID: \"3f849213-a9f5-4236-ba98-a41e70d67dc5\") " Nov 02 18:13:51.784785 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:51.784775 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f849213-a9f5-4236-ba98-a41e70d67dc5-kube-api-access-d9tjm\" (UniqueName: \"kubernetes.io/projected/3f849213-a9f5-4236-ba98-a41e70d67dc5-kube-api-access-d9tjm\") pod \"3f849213-a9f5-4236-ba98-a41e70d67dc5\" (UID: \"3f849213-a9f5-4236-ba98-a41e70d67dc5\") " Nov 02 18:13:51.785160 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:51.785137 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f849213-a9f5-4236-ba98-a41e70d67dc5-openshift-service-ca" pod "3f849213-a9f5-4236-ba98-a41e70d67dc5" (UID: "3f849213-a9f5-4236-ba98-a41e70d67dc5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:13:51.786752 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:51.786729 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f849213-a9f5-4236-ba98-a41e70d67dc5-metrics-certs" pod "3f849213-a9f5-4236-ba98-a41e70d67dc5" (UID: "3f849213-a9f5-4236-ba98-a41e70d67dc5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:13:51.786983 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:51.786949 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f849213-a9f5-4236-ba98-a41e70d67dc5-maas-api-tls" pod "3f849213-a9f5-4236-ba98-a41e70d67dc5" (UID: "3f849213-a9f5-4236-ba98-a41e70d67dc5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:13:51.786983 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:51.786970 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f849213-a9f5-4236-ba98-a41e70d67dc5-kube-api-access-d9tjm" pod "3f849213-a9f5-4236-ba98-a41e70d67dc5" (UID: "3f849213-a9f5-4236-ba98-a41e70d67dc5"). InnerVolumeSpecName "kube-api-access-d9tjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:51.886115 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:51.886080 4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f849213-a9f5-4236-ba98-a41e70d67dc5-metrics-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:51.886115 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:51.886108 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f849213-a9f5-4236-ba98-a41e70d67dc5-maas-api-tls\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:51.886115 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:51.886118 4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f849213-a9f5-4236-ba98-a41e70d67dc5-openshift-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:51.886375 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:51.886130 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9tjm\" (UniqueName: \"kubernetes.io/projected/3f849213-a9f5-4236-ba98-a41e70d67dc5-kube-api-access-d9tjm\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:13:52.107854 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:52.107823 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-2899ab-59796778-drqbh"] Nov 02 18:13:52.120613 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:52.120585 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-2899ab-59796778-drqbh"] Nov 02 18:13:52.441572 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:13:52.441501 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f849213-a9f5-4236-ba98-a41e70d67dc5" path="/var/lib/kubelet/pods/3f849213-a9f5-4236-ba98-a41e70d67dc5/volumes" Nov 02 18:14:06.938919 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:14:06.938837 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:14:09.189380 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:14:09.189336 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:14:26.270003 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:14:26.269967 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:14:39.182017 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:14:39.181978 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:14:42.471266 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:14:42.471229 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:14:45.165242 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:14:45.165208 4471 scope.go:122] "RemoveContainer" containerID="6fcf37e8225599702a9d076fe478baaabea9a6ca56abbc5ff21642096c20a9b3" Nov 02 18:14:45.174370 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:14:45.174341 4471 scope.go:122] "RemoveContainer" containerID="71932218bdacfd0bd3e0827cef38b1fa074e410e5b487855ced20d0b6255042b" Nov 02 18:14:52.621246 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:14:52.621212 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p"] Nov 02 18:14:55.376189 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:14:55.376155 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:15:00.151444 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.151407 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894055-s6jvb"] Nov 02 18:15:00.151916 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.151850 4471 cpu_manager.go:414] "containerMap: removing container" podUID="d2d5291d-d4e9-462b-8f3e-f87f3b82d946" containerName="payload-pre-processing" Nov 02 18:15:00.151916 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.151861 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2d5291d-d4e9-462b-8f3e-f87f3b82d946" containerName="payload-pre-processing" Nov 02 18:15:00.151916 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.151874 4471 cpu_manager.go:414] "containerMap: removing container" podUID="1408655c-5efa-4883-b9d2-0b51327d40bb" containerName="payload-pre-processing" Nov 02 18:15:00.151916 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.151880 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1408655c-5efa-4883-b9d2-0b51327d40bb" containerName="payload-pre-processing" Nov 02 18:15:00.151916 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.151891 4471 cpu_manager.go:414] "containerMap: removing container" podUID="3f849213-a9f5-4236-ba98-a41e70d67dc5" containerName="maas-api" Nov 02 18:15:00.151916 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.151896 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f849213-a9f5-4236-ba98-a41e70d67dc5" containerName="maas-api" Nov 02 18:15:00.151916 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.151903 4471 cpu_manager.go:414] "containerMap: removing container" podUID="29bc04a6-3a7f-4574-9017-ead0d778ac34" containerName="istio-proxy" Nov 02 18:15:00.151916 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.151908 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29bc04a6-3a7f-4574-9017-ead0d778ac34" containerName="istio-proxy" Nov 02 18:15:00.151916 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.151914 4471 cpu_manager.go:414] "containerMap: removing container" podUID="85212099-c279-459d-a468-b868bb55071b" containerName="istio-proxy" Nov 02 18:15:00.151916 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.151919 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85212099-c279-459d-a468-b868bb55071b" containerName="istio-proxy" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.151927 4471 cpu_manager.go:414] "containerMap: removing container" podUID="c987fa61-24a5-40f2-956f-70f3947bdcd7" containerName="istio-proxy" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.151931 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c987fa61-24a5-40f2-956f-70f3947bdcd7" containerName="istio-proxy" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.151939 4471 cpu_manager.go:414] "containerMap: removing container" podUID="0381ed09-c883-448a-baf2-c7cde6ed90ab" containerName="payload-processing" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.151943 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0381ed09-c883-448a-baf2-c7cde6ed90ab" containerName="payload-processing" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.151950 4471 cpu_manager.go:414] "containerMap: removing container" podUID="7395fee0-6fb5-4a90-be40-ba36fde4b163" containerName="payload-pre-processing" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.151954 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7395fee0-6fb5-4a90-be40-ba36fde4b163" containerName="payload-pre-processing" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.151966 4471 cpu_manager.go:414] "containerMap: removing container" podUID="c987fa61-24a5-40f2-956f-70f3947bdcd7" containerName="istio-proxy" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.151970 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c987fa61-24a5-40f2-956f-70f3947bdcd7" containerName="istio-proxy" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.151977 4471 cpu_manager.go:414] "containerMap: removing container" podUID="85212099-c279-459d-a468-b868bb55071b" containerName="istio-proxy" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.151981 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85212099-c279-459d-a468-b868bb55071b" containerName="istio-proxy" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.151987 4471 cpu_manager.go:414] "containerMap: removing container" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.151991 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.151996 4471 cpu_manager.go:414] "containerMap: removing container" podUID="c5bf24bf-3188-425c-bced-4e77490d6377" containerName="maas-api" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152001 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5bf24bf-3188-425c-bced-4e77490d6377" containerName="maas-api" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152007 4471 cpu_manager.go:414] "containerMap: removing container" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152011 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152018 4471 cpu_manager.go:414] "containerMap: removing container" podUID="b54fd98c-e8c7-4578-b027-5275834e8c72" containerName="istio-proxy" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152022 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b54fd98c-e8c7-4578-b027-5275834e8c72" containerName="istio-proxy" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152032 4471 cpu_manager.go:414] "containerMap: removing container" podUID="fae7257d-c6ad-4e86-8133-aacf8110adb5" containerName="istio-proxy" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152036 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fae7257d-c6ad-4e86-8133-aacf8110adb5" containerName="istio-proxy" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152043 4471 cpu_manager.go:414] "containerMap: removing container" podUID="2c047abd-2c48-4b50-ba99-579d0f2cc3fd" containerName="istio-proxy" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152047 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c047abd-2c48-4b50-ba99-579d0f2cc3fd" containerName="istio-proxy" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152056 4471 cpu_manager.go:414] "containerMap: removing container" podUID="c3f2ca77-1f97-44a4-99f1-291daab2dee3" containerName="payload-processing" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152060 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3f2ca77-1f97-44a4-99f1-291daab2dee3" containerName="payload-processing" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152067 4471 cpu_manager.go:414] "containerMap: removing container" podUID="411e5db0-b883-4372-b0ce-42ced3dd0a9b" containerName="maas-api" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152072 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="411e5db0-b883-4372-b0ce-42ced3dd0a9b" containerName="maas-api" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152078 4471 cpu_manager.go:414] "containerMap: removing container" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152083 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152089 4471 cpu_manager.go:414] "containerMap: removing container" podUID="3de46b2c-81e7-458e-b693-088684a8a8d8" containerName="istio-proxy" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152094 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3de46b2c-81e7-458e-b693-088684a8a8d8" containerName="istio-proxy" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152102 4471 cpu_manager.go:414] "containerMap: removing container" podUID="4d5d1db2-0240-442b-aaed-b2931e9e8d6a" containerName="maas-api" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152106 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d5d1db2-0240-442b-aaed-b2931e9e8d6a" containerName="maas-api" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152116 4471 cpu_manager.go:414] "containerMap: removing container" podUID="db524df0-282c-43eb-936b-53ed7d9388cf" containerName="istio-proxy" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152121 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db524df0-282c-43eb-936b-53ed7d9388cf" containerName="istio-proxy" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152129 4471 cpu_manager.go:414] "containerMap: removing container" podUID="e5bb3fbe-93ce-44aa-9cf1-02394cc28af7" containerName="istio-proxy" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152133 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5bb3fbe-93ce-44aa-9cf1-02394cc28af7" containerName="istio-proxy" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152141 4471 cpu_manager.go:414] "containerMap: removing container" podUID="141bdec6-5264-4a27-87e4-699a5bdcf2d7" containerName="maas-api" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152147 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="141bdec6-5264-4a27-87e4-699a5bdcf2d7" containerName="maas-api" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152153 4471 cpu_manager.go:414] "containerMap: removing container" podUID="00044b6e-d520-4934-ba8a-b2d638f899d6" containerName="payload-processing" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152158 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00044b6e-d520-4934-ba8a-b2d638f899d6" containerName="payload-processing" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152214 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="b54fd98c-e8c7-4578-b027-5275834e8c72" containerName="istio-proxy" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152221 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5bb3fbe-93ce-44aa-9cf1-02394cc28af7" containerName="istio-proxy" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152228 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="141bdec6-5264-4a27-87e4-699a5bdcf2d7" containerName="maas-api" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152235 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="00044b6e-d520-4934-ba8a-b2d638f899d6" containerName="payload-processing" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152241 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="3de46b2c-81e7-458e-b693-088684a8a8d8" containerName="istio-proxy" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152246 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2d5291d-d4e9-462b-8f3e-f87f3b82d946" containerName="payload-pre-processing" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152250 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="1408655c-5efa-4883-b9d2-0b51327d40bb" containerName="payload-pre-processing" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152264 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f849213-a9f5-4236-ba98-a41e70d67dc5" containerName="maas-api" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152271 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="29bc04a6-3a7f-4574-9017-ead0d778ac34" containerName="istio-proxy" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152276 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="85212099-c279-459d-a468-b868bb55071b" containerName="istio-proxy" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152281 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="c987fa61-24a5-40f2-956f-70f3947bdcd7" containerName="istio-proxy" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152286 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="0381ed09-c883-448a-baf2-c7cde6ed90ab" containerName="payload-processing" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152292 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="7395fee0-6fb5-4a90-be40-ba36fde4b163" containerName="payload-pre-processing" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152298 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="85212099-c279-459d-a468-b868bb55071b" containerName="istio-proxy" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152316 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c39cbb8-3b30-4f0e-b0f3-4685433cc2d5" containerName="main" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152321 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5bf24bf-3188-425c-bced-4e77490d6377" containerName="maas-api" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152326 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e729ed8-8407-477c-90a7-c899203f2452" containerName="main" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152332 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a819a2f-01ae-45ba-8c60-f3382617c58c" containerName="istio-proxy" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152343 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="fae7257d-c6ad-4e86-8133-aacf8110adb5" containerName="istio-proxy" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152350 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c047abd-2c48-4b50-ba99-579d0f2cc3fd" containerName="istio-proxy" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152356 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3f2ca77-1f97-44a4-99f1-291daab2dee3" containerName="payload-processing" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152363 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="411e5db0-b883-4372-b0ce-42ced3dd0a9b" containerName="maas-api" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152369 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="537a8e6f-da32-4e42-a7a1-bd7c061eabea" containerName="main" Nov 02 18:15:00.152341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152374 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d5d1db2-0240-442b-aaed-b2931e9e8d6a" containerName="maas-api" Nov 02 18:15:00.154708 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.152381 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="db524df0-282c-43eb-936b-53ed7d9388cf" containerName="istio-proxy" Nov 02 18:15:00.157122 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.157104 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894055-s6jvb" Nov 02 18:15:00.190253 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.190224 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 18:15:00.192164 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.192135 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894055-s6jvb"] Nov 02 18:15:00.276797 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.276758 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d21c3fe-6394-409f-86f1-cae34e9b6ee2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894055-s6jvb\" (UID: \"7d21c3fe-6394-409f-86f1-cae34e9b6ee2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894055-s6jvb" Nov 02 18:15:00.276981 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.276844 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xjfh\" (UniqueName: \"kubernetes.io/projected/7d21c3fe-6394-409f-86f1-cae34e9b6ee2-kube-api-access-8xjfh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894055-s6jvb\" (UID: \"7d21c3fe-6394-409f-86f1-cae34e9b6ee2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894055-s6jvb" Nov 02 18:15:00.378208 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.378176 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d21c3fe-6394-409f-86f1-cae34e9b6ee2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894055-s6jvb\" (UID: \"7d21c3fe-6394-409f-86f1-cae34e9b6ee2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894055-s6jvb" Nov 02 18:15:00.378415 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.378244 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xjfh\" (UniqueName: \"kubernetes.io/projected/7d21c3fe-6394-409f-86f1-cae34e9b6ee2-kube-api-access-8xjfh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894055-s6jvb\" (UID: \"7d21c3fe-6394-409f-86f1-cae34e9b6ee2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894055-s6jvb" Nov 02 18:15:00.378914 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.378894 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d21c3fe-6394-409f-86f1-cae34e9b6ee2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894055-s6jvb\" (UID: \"7d21c3fe-6394-409f-86f1-cae34e9b6ee2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894055-s6jvb" Nov 02 18:15:00.402669 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.402614 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xjfh\" (UniqueName: \"kubernetes.io/projected/7d21c3fe-6394-409f-86f1-cae34e9b6ee2-kube-api-access-8xjfh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894055-s6jvb\" (UID: \"7d21c3fe-6394-409f-86f1-cae34e9b6ee2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894055-s6jvb" Nov 02 18:15:00.469283 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.469257 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894055-s6jvb" Nov 02 18:15:00.764463 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:00.764433 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894055-s6jvb"] Nov 02 18:15:00.765515 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:15:00.765488 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d21c3fe_6394_409f_86f1_cae34e9b6ee2.slice/crio-ff393d769044fabb4d14a0852454776dbb1f08a4af6aa542822abde5a699d2cc WatchSource:0}: Error finding container ff393d769044fabb4d14a0852454776dbb1f08a4af6aa542822abde5a699d2cc: Status 404 returned error can't find the container with id ff393d769044fabb4d14a0852454776dbb1f08a4af6aa542822abde5a699d2cc Nov 02 18:15:01.108519 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:01.108486 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894055-s6jvb" event={"ID":"7d21c3fe-6394-409f-86f1-cae34e9b6ee2","Type":"ContainerStarted","Data":"a62c85e17f9442986f30cbe243238b1b3f7c7b70894576706959869816352452"} Nov 02 18:15:01.108674 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:01.108539 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894055-s6jvb" event={"ID":"7d21c3fe-6394-409f-86f1-cae34e9b6ee2","Type":"ContainerStarted","Data":"ff393d769044fabb4d14a0852454776dbb1f08a4af6aa542822abde5a699d2cc"} Nov 02 18:15:01.150126 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:01.150074 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894055-s6jvb" podStartSLOduration=1.1500600460000001 podStartE2EDuration="1.150060046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:01.149327473 +0000 UTC m=+2718.844682939" watchObservedRunningTime="2026-11-02 18:15:01.150060046 +0000 UTC m=+2718.845415519" Nov 02 18:15:02.121907 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:02.121869 4471 generic.go:356] "Generic (PLEG): container finished" podID="7d21c3fe-6394-409f-86f1-cae34e9b6ee2" containerID="a62c85e17f9442986f30cbe243238b1b3f7c7b70894576706959869816352452" exitCode=60 Nov 02 18:15:02.121907 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:02.121920 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894055-s6jvb" event={"ID":"7d21c3fe-6394-409f-86f1-cae34e9b6ee2","Type":"ContainerDied","Data":"a62c85e17f9442986f30cbe243238b1b3f7c7b70894576706959869816352452"} Nov 02 18:15:02.122522 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:02.122235 4471 scope.go:122] "RemoveContainer" containerID="a62c85e17f9442986f30cbe243238b1b3f7c7b70894576706959869816352452" Nov 02 18:15:03.128278 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:03.128191 4471 generic.go:356] "Generic (PLEG): container finished" podID="7d21c3fe-6394-409f-86f1-cae34e9b6ee2" containerID="79d0ee8517b0cdfc80bc0700ba7ed49e7ee91212231ac7f1ef50f4fb5b0a882b" exitCode=60 Nov 02 18:15:03.128666 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:03.128285 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894055-s6jvb" event={"ID":"7d21c3fe-6394-409f-86f1-cae34e9b6ee2","Type":"ContainerDied","Data":"79d0ee8517b0cdfc80bc0700ba7ed49e7ee91212231ac7f1ef50f4fb5b0a882b"} Nov 02 18:15:03.128666 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:03.128366 4471 scope.go:122] "RemoveContainer" containerID="a62c85e17f9442986f30cbe243238b1b3f7c7b70894576706959869816352452" Nov 02 18:15:03.128666 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:03.128659 4471 scope.go:122] "RemoveContainer" containerID="79d0ee8517b0cdfc80bc0700ba7ed49e7ee91212231ac7f1ef50f4fb5b0a882b" Nov 02 18:15:03.129203 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:15:03.128970 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894055-s6jvb_odh-ai-gateway-infra(7d21c3fe-6394-409f-86f1-cae34e9b6ee2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894055-s6jvb" podUID="7d21c3fe-6394-409f-86f1-cae34e9b6ee2" Nov 02 18:15:04.135031 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:04.134998 4471 scope.go:122] "RemoveContainer" containerID="79d0ee8517b0cdfc80bc0700ba7ed49e7ee91212231ac7f1ef50f4fb5b0a882b" Nov 02 18:15:04.135464 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:15:04.135239 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894055-s6jvb_odh-ai-gateway-infra(7d21c3fe-6394-409f-86f1-cae34e9b6ee2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894055-s6jvb" podUID="7d21c3fe-6394-409f-86f1-cae34e9b6ee2" Nov 02 18:15:04.553919 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:04.553879 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:15:05.125986 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:05.125950 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-f91b2d-55449f7d65-nqbd9"] Nov 02 18:15:05.160220 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:05.160191 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f91b2d-66d89575-sdlbq"] Nov 02 18:15:07.641297 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:07.641257 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p" podUID="cd1e4dbc-50ca-4729-9be9-cc3f6e928a48" containerName="main" containerID="cri-o://538312ab8c10efac3f699eecbe58273506314ccab6c5484869f154c0be76c7de" gracePeriod=45 Nov 02 18:15:07.900861 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:07.900800 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p" Nov 02 18:15:08.056773 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:08.056737 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-dshm\") pod \"cd1e4dbc-50ca-4729-9be9-cc3f6e928a48\" (UID: \"cd1e4dbc-50ca-4729-9be9-cc3f6e928a48\") " Nov 02 18:15:08.056974 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:08.056788 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-kube-api-access-8nlr7\" (UniqueName: \"kubernetes.io/projected/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-kube-api-access-8nlr7\") pod \"cd1e4dbc-50ca-4729-9be9-cc3f6e928a48\" (UID: \"cd1e4dbc-50ca-4729-9be9-cc3f6e928a48\") " Nov 02 18:15:08.056974 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:08.056837 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-home\" (UniqueName: \"kubernetes.io/empty-dir/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-home\") pod \"cd1e4dbc-50ca-4729-9be9-cc3f6e928a48\" (UID: \"cd1e4dbc-50ca-4729-9be9-cc3f6e928a48\") " Nov 02 18:15:08.056974 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:08.056869 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-tmp-dir\") pod \"cd1e4dbc-50ca-4729-9be9-cc3f6e928a48\" (UID: \"cd1e4dbc-50ca-4729-9be9-cc3f6e928a48\") " Nov 02 18:15:08.057165 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:08.056975 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-tls-certs\") pod \"cd1e4dbc-50ca-4729-9be9-cc3f6e928a48\" (UID: \"cd1e4dbc-50ca-4729-9be9-cc3f6e928a48\") " Nov 02 18:15:08.057165 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:08.057010 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-model-cache\") pod \"cd1e4dbc-50ca-4729-9be9-cc3f6e928a48\" (UID: \"cd1e4dbc-50ca-4729-9be9-cc3f6e928a48\") " Nov 02 18:15:08.057165 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:08.057128 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-home" pod "cd1e4dbc-50ca-4729-9be9-cc3f6e928a48" (UID: "cd1e4dbc-50ca-4729-9be9-cc3f6e928a48"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:08.057165 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:08.057139 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-tmp-dir" pod "cd1e4dbc-50ca-4729-9be9-cc3f6e928a48" (UID: "cd1e4dbc-50ca-4729-9be9-cc3f6e928a48"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:08.057378 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:08.057329 4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-tmp-dir\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:15:08.057378 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:08.057336 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-model-cache" pod "cd1e4dbc-50ca-4729-9be9-cc3f6e928a48" (UID: "cd1e4dbc-50ca-4729-9be9-cc3f6e928a48"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:08.057378 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:08.057350 4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-home\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:15:08.059045 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:08.059020 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-dshm" pod "cd1e4dbc-50ca-4729-9be9-cc3f6e928a48" (UID: "cd1e4dbc-50ca-4729-9be9-cc3f6e928a48"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:08.059129 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:08.059086 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-kube-api-access-8nlr7" pod "cd1e4dbc-50ca-4729-9be9-cc3f6e928a48" (UID: "cd1e4dbc-50ca-4729-9be9-cc3f6e928a48"). InnerVolumeSpecName "kube-api-access-8nlr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:08.059176 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:08.059131 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-tls-certs" pod "cd1e4dbc-50ca-4729-9be9-cc3f6e928a48" (UID: "cd1e4dbc-50ca-4729-9be9-cc3f6e928a48"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:15:08.155985 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:08.155917 4471 generic.go:356] "Generic (PLEG): container finished" podID="cd1e4dbc-50ca-4729-9be9-cc3f6e928a48" containerID="538312ab8c10efac3f699eecbe58273506314ccab6c5484869f154c0be76c7de" exitCode=0 Nov 02 18:15:08.155985 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:08.155972 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p" Nov 02 18:15:08.156142 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:08.156008 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p" event={"ID":"cd1e4dbc-50ca-4729-9be9-cc3f6e928a48","Type":"ContainerDied","Data":"538312ab8c10efac3f699eecbe58273506314ccab6c5484869f154c0be76c7de"} Nov 02 18:15:08.156142 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:08.156060 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p" event={"ID":"cd1e4dbc-50ca-4729-9be9-cc3f6e928a48","Type":"ContainerDied","Data":"04db58c689a1b469ec9e168e2e4c3605824462209d713963bfd21af13e0d692c"} Nov 02 18:15:08.156142 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:08.156079 4471 scope.go:122] "RemoveContainer" containerID="538312ab8c10efac3f699eecbe58273506314ccab6c5484869f154c0be76c7de" Nov 02 18:15:08.157968 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:08.157941 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nlr7\" (UniqueName: \"kubernetes.io/projected/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-kube-api-access-8nlr7\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:15:08.158060 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:08.157970 4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-tls-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:15:08.158060 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:08.157988 4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-model-cache\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:15:08.158060 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:08.158002 4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48-dshm\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:15:08.165547 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:08.165530 4471 scope.go:122] "RemoveContainer" containerID="538312ab8c10efac3f699eecbe58273506314ccab6c5484869f154c0be76c7de" Nov 02 18:15:08.165842 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:15:08.165822 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"538312ab8c10efac3f699eecbe58273506314ccab6c5484869f154c0be76c7de\": container with ID starting with 538312ab8c10efac3f699eecbe58273506314ccab6c5484869f154c0be76c7de not found: ID does not exist" containerID="538312ab8c10efac3f699eecbe58273506314ccab6c5484869f154c0be76c7de" Nov 02 18:15:08.165898 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:08.165849 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"538312ab8c10efac3f699eecbe58273506314ccab6c5484869f154c0be76c7de"} err="failed to get container status \"538312ab8c10efac3f699eecbe58273506314ccab6c5484869f154c0be76c7de\": rpc error: code = NotFound desc = could not find container \"538312ab8c10efac3f699eecbe58273506314ccab6c5484869f154c0be76c7de\": container with ID starting with 538312ab8c10efac3f699eecbe58273506314ccab6c5484869f154c0be76c7de not found: ID does not exist" Nov 02 18:15:08.208355 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:08.208335 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p"] Nov 02 18:15:08.213800 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:08.213776 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-f91b2d/auth-test-model-f91b2d-kserve-574c85fd65-tdn8p"] Nov 02 18:15:08.439986 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:08.439910 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd1e4dbc-50ca-4729-9be9-cc3f6e928a48" path="/var/lib/kubelet/pods/cd1e4dbc-50ca-4729-9be9-cc3f6e928a48/volumes" Nov 02 18:15:09.183041 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:15:09.183007 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:15:10.126545 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:10.126499 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f91b2d-55449f7d65-nqbd9" podUID="34577a2f-c05a-43ea-8f04-53eedb09fa05" containerName="payload-processing" containerID="cri-o://d2a2cddf9af4db33b5d3e7cd3e37c702a3ef9e1deb08341db52b7efee7575f69" gracePeriod=25 Nov 02 18:15:10.161060 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:10.161023 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f91b2d-66d89575-sdlbq" podUID="6b1fbd6b-0681-4558-aa65-95e14f712af6" containerName="payload-pre-processing" containerID="cri-o://891afef8e7bfd575eca7c01d2aa12528280810d7c0d04ad24a89d8aeb5a9c561" gracePeriod=25 Nov 02 18:15:10.997250 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:10.997217 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894055-s6jvb"] Nov 02 18:15:11.139808 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:11.139784 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894055-s6jvb" Nov 02 18:15:11.172971 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:11.172939 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894055-s6jvb" event={"ID":"7d21c3fe-6394-409f-86f1-cae34e9b6ee2","Type":"ContainerDied","Data":"ff393d769044fabb4d14a0852454776dbb1f08a4af6aa542822abde5a699d2cc"} Nov 02 18:15:11.173138 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:11.172993 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894055-s6jvb" Nov 02 18:15:11.173138 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:11.173000 4471 scope.go:122] "RemoveContainer" containerID="79d0ee8517b0cdfc80bc0700ba7ed49e7ee91212231ac7f1ef50f4fb5b0a882b" Nov 02 18:15:11.191920 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:11.190725 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d21c3fe-6394-409f-86f1-cae34e9b6ee2-kube-api-access-8xjfh\" (UniqueName: \"kubernetes.io/projected/7d21c3fe-6394-409f-86f1-cae34e9b6ee2-kube-api-access-8xjfh\") pod \"7d21c3fe-6394-409f-86f1-cae34e9b6ee2\" (UID: \"7d21c3fe-6394-409f-86f1-cae34e9b6ee2\") " Nov 02 18:15:11.191920 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:11.190800 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d21c3fe-6394-409f-86f1-cae34e9b6ee2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d21c3fe-6394-409f-86f1-cae34e9b6ee2-maas-api-service-ca\") pod \"7d21c3fe-6394-409f-86f1-cae34e9b6ee2\" (UID: \"7d21c3fe-6394-409f-86f1-cae34e9b6ee2\") " Nov 02 18:15:11.192363 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:11.192331 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d21c3fe-6394-409f-86f1-cae34e9b6ee2-maas-api-service-ca" pod "7d21c3fe-6394-409f-86f1-cae34e9b6ee2" (UID: "7d21c3fe-6394-409f-86f1-cae34e9b6ee2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:11.194224 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:11.194196 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d21c3fe-6394-409f-86f1-cae34e9b6ee2-kube-api-access-8xjfh" pod "7d21c3fe-6394-409f-86f1-cae34e9b6ee2" (UID: "7d21c3fe-6394-409f-86f1-cae34e9b6ee2"). InnerVolumeSpecName "kube-api-access-8xjfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:11.291570 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:11.291539 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xjfh\" (UniqueName: \"kubernetes.io/projected/7d21c3fe-6394-409f-86f1-cae34e9b6ee2-kube-api-access-8xjfh\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:15:11.291570 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:11.291565 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d21c3fe-6394-409f-86f1-cae34e9b6ee2-maas-api-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:15:11.518845 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:11.518816 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894055-s6jvb"] Nov 02 18:15:11.532040 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:11.532016 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894055-s6jvb"] Nov 02 18:15:12.447260 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:12.447221 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d21c3fe-6394-409f-86f1-cae34e9b6ee2" path="/var/lib/kubelet/pods/7d21c3fe-6394-409f-86f1-cae34e9b6ee2/volumes" Nov 02 18:15:14.252747 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:14.252710 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:15:17.443401 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:17.443363 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f91b2d-55449f7d65-nqbd9" podUID="34577a2f-c05a-43ea-8f04-53eedb09fa05" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:15:17.443851 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:17.443362 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f91b2d-66d89575-sdlbq" podUID="6b1fbd6b-0681-4558-aa65-95e14f712af6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:15:19.298836 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:19.298804 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658"] Nov 02 18:15:19.299401 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:19.299076 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" podUID="684a3084-e591-4e7c-8e0d-7f2d65211a2a" containerName="istio-proxy" containerID="cri-o://e450181b4a251dc968cbae0bd5daa90c987f0dbf50ae3df36ae55aba977ecb3f" gracePeriod=30 Nov 02 18:15:24.582974 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:24.582943 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 18:15:24.610315 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:24.610287 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istiod-ca-cert\") pod \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " Nov 02 18:15:24.610471 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:24.610345 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-workload-socket\") pod \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " Nov 02 18:15:24.610471 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:24.610387 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istio-podinfo\") pod \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " Nov 02 18:15:24.610471 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:24.610423 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istio-envoy\") pod \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " Nov 02 18:15:24.610471 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:24.610463 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/684a3084-e591-4e7c-8e0d-7f2d65211a2a-kube-api-access-dbs8z\" (UniqueName: \"kubernetes.io/projected/684a3084-e591-4e7c-8e0d-7f2d65211a2a-kube-api-access-dbs8z\") pod \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " Nov 02 18:15:24.610780 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:24.610503 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istio-data\") pod \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " Nov 02 18:15:24.610780 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:24.610558 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-workload-certs\") pod \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " Nov 02 18:15:24.610780 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:24.610617 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istio-token\" (UniqueName: \"kubernetes.io/projected/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istio-token\") pod \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " Nov 02 18:15:24.610780 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:24.610716 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-credential-socket\") pod \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\" (UID: \"684a3084-e591-4e7c-8e0d-7f2d65211a2a\") " Nov 02 18:15:24.611005 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:24.610928 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-workload-socket" pod "684a3084-e591-4e7c-8e0d-7f2d65211a2a" (UID: "684a3084-e591-4e7c-8e0d-7f2d65211a2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:24.611182 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:24.611158 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-credential-socket" pod "684a3084-e591-4e7c-8e0d-7f2d65211a2a" (UID: "684a3084-e591-4e7c-8e0d-7f2d65211a2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:24.611356 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:24.611283 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:15:24.611494 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:24.611467 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istio-data" pod "684a3084-e591-4e7c-8e0d-7f2d65211a2a" (UID: "684a3084-e591-4e7c-8e0d-7f2d65211a2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:24.611494 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:24.611482 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-workload-certs" pod "684a3084-e591-4e7c-8e0d-7f2d65211a2a" (UID: "684a3084-e591-4e7c-8e0d-7f2d65211a2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:24.611622 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:24.611491 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istiod-ca-cert" pod "684a3084-e591-4e7c-8e0d-7f2d65211a2a" (UID: "684a3084-e591-4e7c-8e0d-7f2d65211a2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:24.612928 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:24.612900 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istio-podinfo" pod "684a3084-e591-4e7c-8e0d-7f2d65211a2a" (UID: "684a3084-e591-4e7c-8e0d-7f2d65211a2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:15:24.613145 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:24.613124 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istio-envoy" pod "684a3084-e591-4e7c-8e0d-7f2d65211a2a" (UID: "684a3084-e591-4e7c-8e0d-7f2d65211a2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:24.613237 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:24.613196 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/684a3084-e591-4e7c-8e0d-7f2d65211a2a-kube-api-access-dbs8z" pod "684a3084-e591-4e7c-8e0d-7f2d65211a2a" (UID: "684a3084-e591-4e7c-8e0d-7f2d65211a2a"). InnerVolumeSpecName "kube-api-access-dbs8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:24.613337 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:24.613318 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istio-token" pod "684a3084-e591-4e7c-8e0d-7f2d65211a2a" (UID: "684a3084-e591-4e7c-8e0d-7f2d65211a2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:24.712559 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:24.712521 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbs8z\" (UniqueName: \"kubernetes.io/projected/684a3084-e591-4e7c-8e0d-7f2d65211a2a-kube-api-access-dbs8z\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:15:24.712559 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:24.712555 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:15:24.712763 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:24.712570 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:15:24.712763 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:24.712578 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:15:24.712763 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:24.712586 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:15:24.712763 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:24.712595 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:15:24.712763 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:24.712603 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:15:24.712763 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:24.712610 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/684a3084-e591-4e7c-8e0d-7f2d65211a2a-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:15:25.248676 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:25.248639 4471 generic.go:356] "Generic (PLEG): container finished" podID="684a3084-e591-4e7c-8e0d-7f2d65211a2a" containerID="e450181b4a251dc968cbae0bd5daa90c987f0dbf50ae3df36ae55aba977ecb3f" exitCode=0 Nov 02 18:15:25.248865 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:25.248713 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" event={"ID":"684a3084-e591-4e7c-8e0d-7f2d65211a2a","Type":"ContainerDied","Data":"e450181b4a251dc968cbae0bd5daa90c987f0dbf50ae3df36ae55aba977ecb3f"} Nov 02 18:15:25.248865 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:25.248721 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" Nov 02 18:15:25.248865 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:25.248802 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658" event={"ID":"684a3084-e591-4e7c-8e0d-7f2d65211a2a","Type":"ContainerDied","Data":"37d60d1f59812c750bdc33682cb62601d6b92eaae4ad538829f0ffb75e20f24e"} Nov 02 18:15:25.248865 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:25.248836 4471 scope.go:122] "RemoveContainer" containerID="e450181b4a251dc968cbae0bd5daa90c987f0dbf50ae3df36ae55aba977ecb3f" Nov 02 18:15:25.258757 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:25.258729 4471 scope.go:122] "RemoveContainer" containerID="7a32f19a3d93490aca8b7b3d76882c2fbc5b96cce7557f6cd9b19f62be53f705" Nov 02 18:15:25.267280 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:25.267260 4471 scope.go:122] "RemoveContainer" containerID="e450181b4a251dc968cbae0bd5daa90c987f0dbf50ae3df36ae55aba977ecb3f" Nov 02 18:15:25.267543 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:15:25.267518 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e450181b4a251dc968cbae0bd5daa90c987f0dbf50ae3df36ae55aba977ecb3f\": container with ID starting with e450181b4a251dc968cbae0bd5daa90c987f0dbf50ae3df36ae55aba977ecb3f not found: ID does not exist" containerID="e450181b4a251dc968cbae0bd5daa90c987f0dbf50ae3df36ae55aba977ecb3f" Nov 02 18:15:25.267614 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:25.267552 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e450181b4a251dc968cbae0bd5daa90c987f0dbf50ae3df36ae55aba977ecb3f"} err="failed to get container status \"e450181b4a251dc968cbae0bd5daa90c987f0dbf50ae3df36ae55aba977ecb3f\": rpc error: code = NotFound desc = could not find container \"e450181b4a251dc968cbae0bd5daa90c987f0dbf50ae3df36ae55aba977ecb3f\": container with ID starting with e450181b4a251dc968cbae0bd5daa90c987f0dbf50ae3df36ae55aba977ecb3f not found: ID does not exist" Nov 02 18:15:25.267614 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:25.267571 4471 scope.go:122] "RemoveContainer" containerID="7a32f19a3d93490aca8b7b3d76882c2fbc5b96cce7557f6cd9b19f62be53f705" Nov 02 18:15:25.267847 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:15:25.267828 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a32f19a3d93490aca8b7b3d76882c2fbc5b96cce7557f6cd9b19f62be53f705\": container with ID starting with 7a32f19a3d93490aca8b7b3d76882c2fbc5b96cce7557f6cd9b19f62be53f705 not found: ID does not exist" containerID="7a32f19a3d93490aca8b7b3d76882c2fbc5b96cce7557f6cd9b19f62be53f705" Nov 02 18:15:25.267898 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:25.267853 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a32f19a3d93490aca8b7b3d76882c2fbc5b96cce7557f6cd9b19f62be53f705"} err="failed to get container status \"7a32f19a3d93490aca8b7b3d76882c2fbc5b96cce7557f6cd9b19f62be53f705\": rpc error: code = NotFound desc = could not find container \"7a32f19a3d93490aca8b7b3d76882c2fbc5b96cce7557f6cd9b19f62be53f705\": container with ID starting with 7a32f19a3d93490aca8b7b3d76882c2fbc5b96cce7557f6cd9b19f62be53f705 not found: ID does not exist" Nov 02 18:15:25.314768 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:25.314738 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658"] Nov 02 18:15:25.320112 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:25.320083 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gx658"] Nov 02 18:15:26.439729 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:26.439690 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="684a3084-e591-4e7c-8e0d-7f2d65211a2a" path="/var/lib/kubelet/pods/684a3084-e591-4e7c-8e0d-7f2d65211a2a/volumes" Nov 02 18:15:27.444080 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:27.444044 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f91b2d-55449f7d65-nqbd9" podUID="34577a2f-c05a-43ea-8f04-53eedb09fa05" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:15:27.444080 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:27.444060 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f91b2d-66d89575-sdlbq" podUID="6b1fbd6b-0681-4558-aa65-95e14f712af6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:15:30.288749 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:30.288704 4471 generic.go:356] "Generic (PLEG): container finished" podID="6b1fbd6b-0681-4558-aa65-95e14f712af6" containerID="891afef8e7bfd575eca7c01d2aa12528280810d7c0d04ad24a89d8aeb5a9c561" exitCode=0 Nov 02 18:15:30.289205 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:30.288809 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f91b2d-66d89575-sdlbq" event={"ID":"6b1fbd6b-0681-4558-aa65-95e14f712af6","Type":"ContainerDied","Data":"891afef8e7bfd575eca7c01d2aa12528280810d7c0d04ad24a89d8aeb5a9c561"} Nov 02 18:15:30.290969 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:30.290943 4471 generic.go:356] "Generic (PLEG): container finished" podID="34577a2f-c05a-43ea-8f04-53eedb09fa05" containerID="d2a2cddf9af4db33b5d3e7cd3e37c702a3ef9e1deb08341db52b7efee7575f69" exitCode=0 Nov 02 18:15:30.291095 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:30.291027 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f91b2d-55449f7d65-nqbd9" event={"ID":"34577a2f-c05a-43ea-8f04-53eedb09fa05","Type":"ContainerDied","Data":"d2a2cddf9af4db33b5d3e7cd3e37c702a3ef9e1deb08341db52b7efee7575f69"} Nov 02 18:15:30.341729 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:30.341692 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4dbe3-55667dc67-nnjv8"] Nov 02 18:15:30.428664 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:30.428626 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f91b2d-66d89575-sdlbq" Nov 02 18:15:30.445401 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:30.445377 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f91b2d-55449f7d65-nqbd9" Nov 02 18:15:30.465735 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:30.465703 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b1fbd6b-0681-4558-aa65-95e14f712af6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b1fbd6b-0681-4558-aa65-95e14f712af6-plugins-config-volume\") pod \"6b1fbd6b-0681-4558-aa65-95e14f712af6\" (UID: \"6b1fbd6b-0681-4558-aa65-95e14f712af6\") " Nov 02 18:15:30.465903 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:30.465851 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b1fbd6b-0681-4558-aa65-95e14f712af6-kube-api-access-6pt5c\" (UniqueName: \"kubernetes.io/projected/6b1fbd6b-0681-4558-aa65-95e14f712af6-kube-api-access-6pt5c\") pod \"6b1fbd6b-0681-4558-aa65-95e14f712af6\" (UID: \"6b1fbd6b-0681-4558-aa65-95e14f712af6\") " Nov 02 18:15:30.466044 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:30.466022 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b1fbd6b-0681-4558-aa65-95e14f712af6-plugins-config-volume" pod "6b1fbd6b-0681-4558-aa65-95e14f712af6" (UID: "6b1fbd6b-0681-4558-aa65-95e14f712af6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:30.467881 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:30.467852 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b1fbd6b-0681-4558-aa65-95e14f712af6-kube-api-access-6pt5c" pod "6b1fbd6b-0681-4558-aa65-95e14f712af6" (UID: "6b1fbd6b-0681-4558-aa65-95e14f712af6"). InnerVolumeSpecName "kube-api-access-6pt5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:30.566287 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:30.566250 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34577a2f-c05a-43ea-8f04-53eedb09fa05-kube-api-access-krbds\" (UniqueName: \"kubernetes.io/projected/34577a2f-c05a-43ea-8f04-53eedb09fa05-kube-api-access-krbds\") pod \"34577a2f-c05a-43ea-8f04-53eedb09fa05\" (UID: \"34577a2f-c05a-43ea-8f04-53eedb09fa05\") " Nov 02 18:15:30.566499 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:30.566369 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34577a2f-c05a-43ea-8f04-53eedb09fa05-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34577a2f-c05a-43ea-8f04-53eedb09fa05-plugins-config-volume\") pod \"34577a2f-c05a-43ea-8f04-53eedb09fa05\" (UID: \"34577a2f-c05a-43ea-8f04-53eedb09fa05\") " Nov 02 18:15:30.566637 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:30.566623 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pt5c\" (UniqueName: \"kubernetes.io/projected/6b1fbd6b-0681-4558-aa65-95e14f712af6-kube-api-access-6pt5c\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:15:30.566637 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:30.566637 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b1fbd6b-0681-4558-aa65-95e14f712af6-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:15:30.566794 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:30.566770 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34577a2f-c05a-43ea-8f04-53eedb09fa05-plugins-config-volume" pod "34577a2f-c05a-43ea-8f04-53eedb09fa05" (UID: "34577a2f-c05a-43ea-8f04-53eedb09fa05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:30.568349 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:30.568319 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34577a2f-c05a-43ea-8f04-53eedb09fa05-kube-api-access-krbds" pod "34577a2f-c05a-43ea-8f04-53eedb09fa05" (UID: "34577a2f-c05a-43ea-8f04-53eedb09fa05"). InnerVolumeSpecName "kube-api-access-krbds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:30.668288 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:30.668204 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krbds\" (UniqueName: \"kubernetes.io/projected/34577a2f-c05a-43ea-8f04-53eedb09fa05-kube-api-access-krbds\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:15:30.668288 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:30.668240 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34577a2f-c05a-43ea-8f04-53eedb09fa05-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:15:31.297052 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:31.297026 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f91b2d-66d89575-sdlbq" Nov 02 18:15:31.297583 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:31.297030 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f91b2d-66d89575-sdlbq" event={"ID":"6b1fbd6b-0681-4558-aa65-95e14f712af6","Type":"ContainerDied","Data":"c3010602331765eefcb82b20487cf8ace52cad3f9579120612572930643cb421"} Nov 02 18:15:31.297583 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:31.297168 4471 scope.go:122] "RemoveContainer" containerID="891afef8e7bfd575eca7c01d2aa12528280810d7c0d04ad24a89d8aeb5a9c561" Nov 02 18:15:31.298501 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:31.298424 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f91b2d-55449f7d65-nqbd9" Nov 02 18:15:31.298501 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:31.298432 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f91b2d-55449f7d65-nqbd9" event={"ID":"34577a2f-c05a-43ea-8f04-53eedb09fa05","Type":"ContainerDied","Data":"d192c51aee769e695e03c5506a590f46a94220a1382ded52156ca37d73fc7908"} Nov 02 18:15:31.306397 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:31.306379 4471 scope.go:122] "RemoveContainer" containerID="d2a2cddf9af4db33b5d3e7cd3e37c702a3ef9e1deb08341db52b7efee7575f69" Nov 02 18:15:31.349462 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:31.349437 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-f91b2d-55449f7d65-nqbd9"] Nov 02 18:15:31.361851 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:31.361827 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-f91b2d-55449f7d65-nqbd9"] Nov 02 18:15:31.387795 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:31.387772 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f91b2d-66d89575-sdlbq"] Nov 02 18:15:31.391757 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:31.391737 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f91b2d-66d89575-sdlbq"] Nov 02 18:15:32.439185 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:32.439151 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34577a2f-c05a-43ea-8f04-53eedb09fa05" path="/var/lib/kubelet/pods/34577a2f-c05a-43ea-8f04-53eedb09fa05/volumes" Nov 02 18:15:32.439566 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:32.439529 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b1fbd6b-0681-4558-aa65-95e14f712af6" path="/var/lib/kubelet/pods/6b1fbd6b-0681-4558-aa65-95e14f712af6/volumes" Nov 02 18:15:35.342408 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:35.342370 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4dbe3-55667dc67-nnjv8" podUID="17058cd8-b4ee-4466-807c-e47b5f98658c" containerName="payload-pre-processing" containerID="cri-o://6ee8e9634caafdf7a729e9927a186ca34cd0e0f336cb81ef656237d228c17d59" gracePeriod=25 Nov 02 18:15:36.489453 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:36.489418 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:15:39.182565 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:15:39.182521 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:15:41.225045 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:41.225014 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk"] Nov 02 18:15:41.225496 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:41.225215 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" podUID="ed3b7de6-bb28-433c-b507-2724a0d524df" containerName="istio-proxy" containerID="cri-o://f5b97eb650650b3a76a709d7952ecede904fbc9d15ddb7fd5a7ca1f61e3e4a41" gracePeriod=30 Nov 02 18:15:41.349442 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:41.349403 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4dbe3-55667dc67-nnjv8" podUID="17058cd8-b4ee-4466-807c-e47b5f98658c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:15:43.292859 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:43.292822 4471 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:15:43.293231 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:43.292913 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" podUID="ed3b7de6-bb28-433c-b507-2724a0d524df" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:15:45.201570 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:45.201536 4471 scope.go:122] "RemoveContainer" containerID="0edbd470905a8279d7c16310d64cd1451ad049249bcf3994b2baaaf793e426f2" Nov 02 18:15:46.374705 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:46.374672 4471 generic.go:356] "Generic (PLEG): container finished" podID="ed3b7de6-bb28-433c-b507-2724a0d524df" containerID="f5b97eb650650b3a76a709d7952ecede904fbc9d15ddb7fd5a7ca1f61e3e4a41" exitCode=0 Nov 02 18:15:46.375097 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:46.374742 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" event={"ID":"ed3b7de6-bb28-433c-b507-2724a0d524df","Type":"ContainerDied","Data":"f5b97eb650650b3a76a709d7952ecede904fbc9d15ddb7fd5a7ca1f61e3e4a41"} Nov 02 18:15:46.375097 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:46.374813 4471 scope.go:122] "RemoveContainer" containerID="14a17f17f0639ef028abb3ff9bfc09c32e1782f4345f288ebeebbebc4c78bd31" Nov 02 18:15:46.477883 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:46.477855 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 18:15:46.620845 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:46.620813 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-workload-socket\") pod \"ed3b7de6-bb28-433c-b507-2724a0d524df\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " Nov 02 18:15:46.621032 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:46.620876 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed3b7de6-bb28-433c-b507-2724a0d524df-kube-api-access-s6vrs\" (UniqueName: \"kubernetes.io/projected/ed3b7de6-bb28-433c-b507-2724a0d524df-kube-api-access-s6vrs\") pod \"ed3b7de6-bb28-433c-b507-2724a0d524df\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " Nov 02 18:15:46.621032 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:46.620903 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed3b7de6-bb28-433c-b507-2724a0d524df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed3b7de6-bb28-433c-b507-2724a0d524df-istio-podinfo\") pod \"ed3b7de6-bb28-433c-b507-2724a0d524df\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " Nov 02 18:15:46.621032 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:46.620931 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-istio-envoy\") pod \"ed3b7de6-bb28-433c-b507-2724a0d524df\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " Nov 02 18:15:46.621032 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:46.620948 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-istio-data\") pod \"ed3b7de6-bb28-433c-b507-2724a0d524df\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " Nov 02 18:15:46.621032 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:46.620980 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed3b7de6-bb28-433c-b507-2724a0d524df-istio-token\" (UniqueName: \"kubernetes.io/projected/ed3b7de6-bb28-433c-b507-2724a0d524df-istio-token\") pod \"ed3b7de6-bb28-433c-b507-2724a0d524df\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " Nov 02 18:15:46.621032 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:46.621014 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed3b7de6-bb28-433c-b507-2724a0d524df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed3b7de6-bb28-433c-b507-2724a0d524df-istiod-ca-cert\") pod \"ed3b7de6-bb28-433c-b507-2724a0d524df\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " Nov 02 18:15:46.621369 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:46.621104 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-credential-socket\") pod \"ed3b7de6-bb28-433c-b507-2724a0d524df\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " Nov 02 18:15:46.621369 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:46.621115 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-workload-socket" pod "ed3b7de6-bb28-433c-b507-2724a0d524df" (UID: "ed3b7de6-bb28-433c-b507-2724a0d524df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:46.621369 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:46.621142 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-workload-certs\") pod \"ed3b7de6-bb28-433c-b507-2724a0d524df\" (UID: \"ed3b7de6-bb28-433c-b507-2724a0d524df\") " Nov 02 18:15:46.621549 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:46.621414 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-istio-data" pod "ed3b7de6-bb28-433c-b507-2724a0d524df" (UID: "ed3b7de6-bb28-433c-b507-2724a0d524df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:46.621549 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:46.621429 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-credential-socket" pod "ed3b7de6-bb28-433c-b507-2724a0d524df" (UID: "ed3b7de6-bb28-433c-b507-2724a0d524df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:46.621645 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:46.621576 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:15:46.621645 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:46.621598 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:15:46.621645 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:46.621613 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:15:46.621645 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:46.621635 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-workload-certs" pod "ed3b7de6-bb28-433c-b507-2724a0d524df" (UID: "ed3b7de6-bb28-433c-b507-2724a0d524df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:46.621808 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:46.621574 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed3b7de6-bb28-433c-b507-2724a0d524df-istiod-ca-cert" pod "ed3b7de6-bb28-433c-b507-2724a0d524df" (UID: "ed3b7de6-bb28-433c-b507-2724a0d524df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:46.623231 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:46.623210 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed3b7de6-bb28-433c-b507-2724a0d524df-kube-api-access-s6vrs" pod "ed3b7de6-bb28-433c-b507-2724a0d524df" (UID: "ed3b7de6-bb28-433c-b507-2724a0d524df"). InnerVolumeSpecName "kube-api-access-s6vrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:46.623468 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:46.623443 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed3b7de6-bb28-433c-b507-2724a0d524df-istio-token" pod "ed3b7de6-bb28-433c-b507-2724a0d524df" (UID: "ed3b7de6-bb28-433c-b507-2724a0d524df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:46.623669 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:46.623651 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-istio-envoy" pod "ed3b7de6-bb28-433c-b507-2724a0d524df" (UID: "ed3b7de6-bb28-433c-b507-2724a0d524df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:46.623740 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:46.623679 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed3b7de6-bb28-433c-b507-2724a0d524df-istio-podinfo" pod "ed3b7de6-bb28-433c-b507-2724a0d524df" (UID: "ed3b7de6-bb28-433c-b507-2724a0d524df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:15:46.722322 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:46.722275 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:15:46.722322 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:46.722299 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6vrs\" (UniqueName: \"kubernetes.io/projected/ed3b7de6-bb28-433c-b507-2724a0d524df-kube-api-access-s6vrs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:15:46.722556 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:46.722336 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed3b7de6-bb28-433c-b507-2724a0d524df-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:15:46.722556 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:46.722345 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed3b7de6-bb28-433c-b507-2724a0d524df-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:15:46.722556 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:46.722353 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed3b7de6-bb28-433c-b507-2724a0d524df-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:15:46.722556 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:46.722361 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed3b7de6-bb28-433c-b507-2724a0d524df-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:15:47.380712 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:47.380675 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" Nov 02 18:15:47.381181 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:47.380710 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk" event={"ID":"ed3b7de6-bb28-433c-b507-2724a0d524df","Type":"ContainerDied","Data":"ac4300ba8c86f5039d100beed124d5683ddeecacc471fde2580b2409eee8057e"} Nov 02 18:15:47.381181 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:47.380773 4471 scope.go:122] "RemoveContainer" containerID="f5b97eb650650b3a76a709d7952ecede904fbc9d15ddb7fd5a7ca1f61e3e4a41" Nov 02 18:15:47.445849 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:47.445826 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk"] Nov 02 18:15:47.458656 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:47.458634 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c4dbe3-openshift-default-67fd9d74b7-d4cqk"] Nov 02 18:15:48.439173 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:48.439137 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed3b7de6-bb28-433c-b507-2724a0d524df" path="/var/lib/kubelet/pods/ed3b7de6-bb28-433c-b507-2724a0d524df/volumes" Nov 02 18:15:51.350043 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:51.350005 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4dbe3-55667dc67-nnjv8" podUID="17058cd8-b4ee-4466-807c-e47b5f98658c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:15:55.589393 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:55.589368 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4dbe3-55667dc67-nnjv8" Nov 02 18:15:55.604398 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:55.604333 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17058cd8-b4ee-4466-807c-e47b5f98658c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17058cd8-b4ee-4466-807c-e47b5f98658c-plugins-config-volume\") pod \"17058cd8-b4ee-4466-807c-e47b5f98658c\" (UID: \"17058cd8-b4ee-4466-807c-e47b5f98658c\") " Nov 02 18:15:55.604398 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:55.604380 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17058cd8-b4ee-4466-807c-e47b5f98658c-kube-api-access-kgkrd\" (UniqueName: \"kubernetes.io/projected/17058cd8-b4ee-4466-807c-e47b5f98658c-kube-api-access-kgkrd\") pod \"17058cd8-b4ee-4466-807c-e47b5f98658c\" (UID: \"17058cd8-b4ee-4466-807c-e47b5f98658c\") " Nov 02 18:15:55.604666 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:55.604644 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17058cd8-b4ee-4466-807c-e47b5f98658c-plugins-config-volume" pod "17058cd8-b4ee-4466-807c-e47b5f98658c" (UID: "17058cd8-b4ee-4466-807c-e47b5f98658c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:55.606329 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:55.606293 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17058cd8-b4ee-4466-807c-e47b5f98658c-kube-api-access-kgkrd" pod "17058cd8-b4ee-4466-807c-e47b5f98658c" (UID: "17058cd8-b4ee-4466-807c-e47b5f98658c"). InnerVolumeSpecName "kube-api-access-kgkrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:55.705631 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:55.705599 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgkrd\" (UniqueName: \"kubernetes.io/projected/17058cd8-b4ee-4466-807c-e47b5f98658c-kube-api-access-kgkrd\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:15:55.705631 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:55.705624 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17058cd8-b4ee-4466-807c-e47b5f98658c-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:15:56.425081 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:56.425040 4471 generic.go:356] "Generic (PLEG): container finished" podID="17058cd8-b4ee-4466-807c-e47b5f98658c" containerID="6ee8e9634caafdf7a729e9927a186ca34cd0e0f336cb81ef656237d228c17d59" exitCode=0 Nov 02 18:15:56.425268 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:56.425093 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4dbe3-55667dc67-nnjv8" Nov 02 18:15:56.425268 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:56.425108 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4dbe3-55667dc67-nnjv8" event={"ID":"17058cd8-b4ee-4466-807c-e47b5f98658c","Type":"ContainerDied","Data":"6ee8e9634caafdf7a729e9927a186ca34cd0e0f336cb81ef656237d228c17d59"} Nov 02 18:15:56.425268 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:56.425161 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4dbe3-55667dc67-nnjv8" event={"ID":"17058cd8-b4ee-4466-807c-e47b5f98658c","Type":"ContainerDied","Data":"3aeeeb962c3f9e23b97a859624e905ffdac6282d434dfc4b7472dd42ba58486b"} Nov 02 18:15:56.425268 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:56.425190 4471 scope.go:122] "RemoveContainer" containerID="6ee8e9634caafdf7a729e9927a186ca34cd0e0f336cb81ef656237d228c17d59" Nov 02 18:15:56.440399 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:56.440373 4471 scope.go:122] "RemoveContainer" containerID="6ee8e9634caafdf7a729e9927a186ca34cd0e0f336cb81ef656237d228c17d59" Nov 02 18:15:56.440743 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:15:56.440692 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ee8e9634caafdf7a729e9927a186ca34cd0e0f336cb81ef656237d228c17d59\": container with ID starting with 6ee8e9634caafdf7a729e9927a186ca34cd0e0f336cb81ef656237d228c17d59 not found: ID does not exist" containerID="6ee8e9634caafdf7a729e9927a186ca34cd0e0f336cb81ef656237d228c17d59" Nov 02 18:15:56.440802 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:56.440755 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee8e9634caafdf7a729e9927a186ca34cd0e0f336cb81ef656237d228c17d59"} err="failed to get container status \"6ee8e9634caafdf7a729e9927a186ca34cd0e0f336cb81ef656237d228c17d59\": rpc error: code = NotFound desc = could not find container \"6ee8e9634caafdf7a729e9927a186ca34cd0e0f336cb81ef656237d228c17d59\": container with ID starting with 6ee8e9634caafdf7a729e9927a186ca34cd0e0f336cb81ef656237d228c17d59 not found: ID does not exist" Nov 02 18:15:56.477447 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:56.477415 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4dbe3-55667dc67-nnjv8"] Nov 02 18:15:56.490216 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:56.490190 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c4dbe3-55667dc67-nnjv8"] Nov 02 18:15:58.438918 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:15:58.438886 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17058cd8-b4ee-4466-807c-e47b5f98658c" path="/var/lib/kubelet/pods/17058cd8-b4ee-4466-807c-e47b5f98658c/volumes" Nov 02 18:16:07.883461 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:07.883416 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:16:09.182457 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:16:09.182423 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:16:10.028507 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:10.028470 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4"] Nov 02 18:16:10.028742 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:10.028711 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4" podUID="d53fefb9-19f7-4b4b-adec-75325b8ff619" containerName="maas-api" containerID="cri-o://094cf74c47a211c8eccf7301f8518af875010b7dc3ba3b08b2077e4ac2050133" gracePeriod=30 Nov 02 18:16:10.277912 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:10.277880 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4" Nov 02 18:16:10.315358 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:10.315284 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d53fefb9-19f7-4b4b-adec-75325b8ff619-kube-api-access-f5pdk\" (UniqueName: \"kubernetes.io/projected/d53fefb9-19f7-4b4b-adec-75325b8ff619-kube-api-access-f5pdk\") pod \"d53fefb9-19f7-4b4b-adec-75325b8ff619\" (UID: \"d53fefb9-19f7-4b4b-adec-75325b8ff619\") " Nov 02 18:16:10.315358 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:10.315353 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d53fefb9-19f7-4b4b-adec-75325b8ff619-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d53fefb9-19f7-4b4b-adec-75325b8ff619-openshift-service-ca\") pod \"d53fefb9-19f7-4b4b-adec-75325b8ff619\" (UID: \"d53fefb9-19f7-4b4b-adec-75325b8ff619\") " Nov 02 18:16:10.315561 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:10.315405 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d53fefb9-19f7-4b4b-adec-75325b8ff619-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d53fefb9-19f7-4b4b-adec-75325b8ff619-maas-api-tls\") pod \"d53fefb9-19f7-4b4b-adec-75325b8ff619\" (UID: \"d53fefb9-19f7-4b4b-adec-75325b8ff619\") " Nov 02 18:16:10.315561 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:10.315449 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d53fefb9-19f7-4b4b-adec-75325b8ff619-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d53fefb9-19f7-4b4b-adec-75325b8ff619-metrics-certs\") pod \"d53fefb9-19f7-4b4b-adec-75325b8ff619\" (UID: \"d53fefb9-19f7-4b4b-adec-75325b8ff619\") " Nov 02 18:16:10.315739 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:10.315716 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d53fefb9-19f7-4b4b-adec-75325b8ff619-openshift-service-ca" pod "d53fefb9-19f7-4b4b-adec-75325b8ff619" (UID: "d53fefb9-19f7-4b4b-adec-75325b8ff619"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:16:10.317575 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:10.317546 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d53fefb9-19f7-4b4b-adec-75325b8ff619-kube-api-access-f5pdk" pod "d53fefb9-19f7-4b4b-adec-75325b8ff619" (UID: "d53fefb9-19f7-4b4b-adec-75325b8ff619"). InnerVolumeSpecName "kube-api-access-f5pdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:16:10.317691 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:10.317661 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d53fefb9-19f7-4b4b-adec-75325b8ff619-maas-api-tls" pod "d53fefb9-19f7-4b4b-adec-75325b8ff619" (UID: "d53fefb9-19f7-4b4b-adec-75325b8ff619"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:16:10.317745 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:10.317730 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d53fefb9-19f7-4b4b-adec-75325b8ff619-metrics-certs" pod "d53fefb9-19f7-4b4b-adec-75325b8ff619" (UID: "d53fefb9-19f7-4b4b-adec-75325b8ff619"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:16:10.416320 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:10.416284 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d53fefb9-19f7-4b4b-adec-75325b8ff619-maas-api-tls\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:16:10.416467 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:10.416337 4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d53fefb9-19f7-4b4b-adec-75325b8ff619-metrics-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:16:10.416467 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:10.416349 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5pdk\" (UniqueName: \"kubernetes.io/projected/d53fefb9-19f7-4b4b-adec-75325b8ff619-kube-api-access-f5pdk\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:16:10.416467 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:10.416357 4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d53fefb9-19f7-4b4b-adec-75325b8ff619-openshift-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:16:10.492025 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:10.491986 4471 generic.go:356] "Generic (PLEG): container finished" podID="d53fefb9-19f7-4b4b-adec-75325b8ff619" containerID="094cf74c47a211c8eccf7301f8518af875010b7dc3ba3b08b2077e4ac2050133" exitCode=0 Nov 02 18:16:10.492215 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:10.492049 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4" Nov 02 18:16:10.492215 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:10.492073 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4" event={"ID":"d53fefb9-19f7-4b4b-adec-75325b8ff619","Type":"ContainerDied","Data":"094cf74c47a211c8eccf7301f8518af875010b7dc3ba3b08b2077e4ac2050133"} Nov 02 18:16:10.492215 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:10.492141 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4" event={"ID":"d53fefb9-19f7-4b4b-adec-75325b8ff619","Type":"ContainerDied","Data":"38f51a3faadf1b82c43f6d2c66f479bb68031bce60c3f3f9d154556d0adfdeae"} Nov 02 18:16:10.492215 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:10.492171 4471 scope.go:122] "RemoveContainer" containerID="094cf74c47a211c8eccf7301f8518af875010b7dc3ba3b08b2077e4ac2050133" Nov 02 18:16:10.501205 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:10.501183 4471 scope.go:122] "RemoveContainer" containerID="094cf74c47a211c8eccf7301f8518af875010b7dc3ba3b08b2077e4ac2050133" Nov 02 18:16:10.501502 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:16:10.501480 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"094cf74c47a211c8eccf7301f8518af875010b7dc3ba3b08b2077e4ac2050133\": container with ID starting with 094cf74c47a211c8eccf7301f8518af875010b7dc3ba3b08b2077e4ac2050133 not found: ID does not exist" containerID="094cf74c47a211c8eccf7301f8518af875010b7dc3ba3b08b2077e4ac2050133" Nov 02 18:16:10.501569 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:10.501509 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"094cf74c47a211c8eccf7301f8518af875010b7dc3ba3b08b2077e4ac2050133"} err="failed to get container status \"094cf74c47a211c8eccf7301f8518af875010b7dc3ba3b08b2077e4ac2050133\": rpc error: code = NotFound desc = could not find container \"094cf74c47a211c8eccf7301f8518af875010b7dc3ba3b08b2077e4ac2050133\": container with ID starting with 094cf74c47a211c8eccf7301f8518af875010b7dc3ba3b08b2077e4ac2050133 not found: ID does not exist" Nov 02 18:16:10.544778 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:10.544752 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4"] Nov 02 18:16:10.548777 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:10.548755 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-069fa5-7f6798f46b-zpgq4"] Nov 02 18:16:12.439188 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:12.439154 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d53fefb9-19f7-4b4b-adec-75325b8ff619" path="/var/lib/kubelet/pods/d53fefb9-19f7-4b4b-adec-75325b8ff619/volumes" Nov 02 18:16:16.535840 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:16.535806 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:16:25.681896 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.681857 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-67b9f44df6-ncjk9"] Nov 02 18:16:25.682435 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682316 4471 cpu_manager.go:414] "containerMap: removing container" podUID="7d21c3fe-6394-409f-86f1-cae34e9b6ee2" containerName="cleanup" Nov 02 18:16:25.682435 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682334 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d21c3fe-6394-409f-86f1-cae34e9b6ee2" containerName="cleanup" Nov 02 18:16:25.682435 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682352 4471 cpu_manager.go:414] "containerMap: removing container" podUID="17058cd8-b4ee-4466-807c-e47b5f98658c" containerName="payload-pre-processing" Nov 02 18:16:25.682435 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682359 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17058cd8-b4ee-4466-807c-e47b5f98658c" containerName="payload-pre-processing" Nov 02 18:16:25.682435 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682374 4471 cpu_manager.go:414] "containerMap: removing container" podUID="3de46b2c-81e7-458e-b693-088684a8a8d8" containerName="istio-proxy" Nov 02 18:16:25.682435 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682379 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3de46b2c-81e7-458e-b693-088684a8a8d8" containerName="istio-proxy" Nov 02 18:16:25.682435 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682389 4471 cpu_manager.go:414] "containerMap: removing container" podUID="ed3b7de6-bb28-433c-b507-2724a0d524df" containerName="istio-proxy" Nov 02 18:16:25.682435 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682398 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed3b7de6-bb28-433c-b507-2724a0d524df" containerName="istio-proxy" Nov 02 18:16:25.682435 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682407 4471 cpu_manager.go:414] "containerMap: removing container" podUID="ed3b7de6-bb28-433c-b507-2724a0d524df" containerName="istio-proxy" Nov 02 18:16:25.682435 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682413 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed3b7de6-bb28-433c-b507-2724a0d524df" containerName="istio-proxy" Nov 02 18:16:25.682435 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682425 4471 cpu_manager.go:414] "containerMap: removing container" podUID="34577a2f-c05a-43ea-8f04-53eedb09fa05" containerName="payload-processing" Nov 02 18:16:25.682435 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682430 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34577a2f-c05a-43ea-8f04-53eedb09fa05" containerName="payload-processing" Nov 02 18:16:25.682435 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682437 4471 cpu_manager.go:414] "containerMap: removing container" podUID="d53fefb9-19f7-4b4b-adec-75325b8ff619" containerName="maas-api" Nov 02 18:16:25.682435 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682442 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d53fefb9-19f7-4b4b-adec-75325b8ff619" containerName="maas-api" Nov 02 18:16:25.683151 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682449 4471 cpu_manager.go:414] "containerMap: removing container" podUID="684a3084-e591-4e7c-8e0d-7f2d65211a2a" containerName="istio-proxy" Nov 02 18:16:25.683151 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682454 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="684a3084-e591-4e7c-8e0d-7f2d65211a2a" containerName="istio-proxy" Nov 02 18:16:25.683151 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682466 4471 cpu_manager.go:414] "containerMap: removing container" podUID="6b1fbd6b-0681-4558-aa65-95e14f712af6" containerName="payload-pre-processing" Nov 02 18:16:25.683151 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682475 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b1fbd6b-0681-4558-aa65-95e14f712af6" containerName="payload-pre-processing" Nov 02 18:16:25.683151 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682486 4471 cpu_manager.go:414] "containerMap: removing container" podUID="cd1e4dbc-50ca-4729-9be9-cc3f6e928a48" containerName="main" Nov 02 18:16:25.683151 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682559 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd1e4dbc-50ca-4729-9be9-cc3f6e928a48" containerName="main" Nov 02 18:16:25.683151 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682578 4471 cpu_manager.go:414] "containerMap: removing container" podUID="684a3084-e591-4e7c-8e0d-7f2d65211a2a" containerName="istio-proxy" Nov 02 18:16:25.683151 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682583 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="684a3084-e591-4e7c-8e0d-7f2d65211a2a" containerName="istio-proxy" Nov 02 18:16:25.683151 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682599 4471 cpu_manager.go:414] "containerMap: removing container" podUID="b54fd98c-e8c7-4578-b027-5275834e8c72" containerName="istio-proxy" Nov 02 18:16:25.683151 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682603 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b54fd98c-e8c7-4578-b027-5275834e8c72" containerName="istio-proxy" Nov 02 18:16:25.683151 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682746 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd1e4dbc-50ca-4729-9be9-cc3f6e928a48" containerName="main" Nov 02 18:16:25.683151 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682765 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="c987fa61-24a5-40f2-956f-70f3947bdcd7" containerName="istio-proxy" Nov 02 18:16:25.683151 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682810 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="684a3084-e591-4e7c-8e0d-7f2d65211a2a" containerName="istio-proxy" Nov 02 18:16:25.683151 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682825 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d21c3fe-6394-409f-86f1-cae34e9b6ee2" containerName="cleanup" Nov 02 18:16:25.683151 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682835 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="17058cd8-b4ee-4466-807c-e47b5f98658c" containerName="payload-pre-processing" Nov 02 18:16:25.683151 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682846 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed3b7de6-bb28-433c-b507-2724a0d524df" containerName="istio-proxy" Nov 02 18:16:25.683151 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682853 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed3b7de6-bb28-433c-b507-2724a0d524df" containerName="istio-proxy" Nov 02 18:16:25.683151 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682864 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="34577a2f-c05a-43ea-8f04-53eedb09fa05" containerName="payload-processing" Nov 02 18:16:25.683151 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682892 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="d53fefb9-19f7-4b4b-adec-75325b8ff619" containerName="maas-api" Nov 02 18:16:25.683151 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682901 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="b54fd98c-e8c7-4578-b027-5275834e8c72" containerName="istio-proxy" Nov 02 18:16:25.683151 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682911 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="3de46b2c-81e7-458e-b693-088684a8a8d8" containerName="istio-proxy" Nov 02 18:16:25.683151 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.682920 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b1fbd6b-0681-4558-aa65-95e14f712af6" containerName="payload-pre-processing" Nov 02 18:16:25.686595 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.686579 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67b9f44df6-ncjk9" Nov 02 18:16:25.714492 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.714466 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 18:16:25.714758 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.714741 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kv77t\"" Nov 02 18:16:25.714758 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.714752 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 18:16:25.714924 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.714813 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 18:16:25.716642 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.716620 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67b9f44df6-ncjk9"] Nov 02 18:16:25.756721 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.756698 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57d9041a-7a4a-4179-bcc2-05d21991cbef-webhook-cert\") pod \"maas-controller-67b9f44df6-ncjk9\" (UID: \"57d9041a-7a4a-4179-bcc2-05d21991cbef\") " pod="opendatahub/maas-controller-67b9f44df6-ncjk9" Nov 02 18:16:25.756838 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.756736 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57d9041a-7a4a-4179-bcc2-05d21991cbef-metrics-certs\") pod \"maas-controller-67b9f44df6-ncjk9\" (UID: \"57d9041a-7a4a-4179-bcc2-05d21991cbef\") " pod="opendatahub/maas-controller-67b9f44df6-ncjk9" Nov 02 18:16:25.756838 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.756820 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl2hf\" (UniqueName: \"kubernetes.io/projected/57d9041a-7a4a-4179-bcc2-05d21991cbef-kube-api-access-xl2hf\") pod \"maas-controller-67b9f44df6-ncjk9\" (UID: \"57d9041a-7a4a-4179-bcc2-05d21991cbef\") " pod="opendatahub/maas-controller-67b9f44df6-ncjk9" Nov 02 18:16:25.857437 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.857405 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57d9041a-7a4a-4179-bcc2-05d21991cbef-webhook-cert\") pod \"maas-controller-67b9f44df6-ncjk9\" (UID: \"57d9041a-7a4a-4179-bcc2-05d21991cbef\") " pod="opendatahub/maas-controller-67b9f44df6-ncjk9" Nov 02 18:16:25.857585 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.857449 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57d9041a-7a4a-4179-bcc2-05d21991cbef-metrics-certs\") pod \"maas-controller-67b9f44df6-ncjk9\" (UID: \"57d9041a-7a4a-4179-bcc2-05d21991cbef\") " pod="opendatahub/maas-controller-67b9f44df6-ncjk9" Nov 02 18:16:25.857585 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.857492 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl2hf\" (UniqueName: \"kubernetes.io/projected/57d9041a-7a4a-4179-bcc2-05d21991cbef-kube-api-access-xl2hf\") pod \"maas-controller-67b9f44df6-ncjk9\" (UID: \"57d9041a-7a4a-4179-bcc2-05d21991cbef\") " pod="opendatahub/maas-controller-67b9f44df6-ncjk9" Nov 02 18:16:25.859995 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.859974 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57d9041a-7a4a-4179-bcc2-05d21991cbef-webhook-cert\") pod \"maas-controller-67b9f44df6-ncjk9\" (UID: \"57d9041a-7a4a-4179-bcc2-05d21991cbef\") " pod="opendatahub/maas-controller-67b9f44df6-ncjk9" Nov 02 18:16:25.859995 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.859983 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57d9041a-7a4a-4179-bcc2-05d21991cbef-metrics-certs\") pod \"maas-controller-67b9f44df6-ncjk9\" (UID: \"57d9041a-7a4a-4179-bcc2-05d21991cbef\") " pod="opendatahub/maas-controller-67b9f44df6-ncjk9" Nov 02 18:16:25.881588 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.881564 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl2hf\" (UniqueName: \"kubernetes.io/projected/57d9041a-7a4a-4179-bcc2-05d21991cbef-kube-api-access-xl2hf\") pod \"maas-controller-67b9f44df6-ncjk9\" (UID: \"57d9041a-7a4a-4179-bcc2-05d21991cbef\") " pod="opendatahub/maas-controller-67b9f44df6-ncjk9" Nov 02 18:16:25.999835 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:25.999756 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67b9f44df6-ncjk9" Nov 02 18:16:26.295276 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:26.295242 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67b9f44df6-ncjk9"] Nov 02 18:16:26.295748 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:16:26.295727 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57d9041a_7a4a_4179_bcc2_05d21991cbef.slice/crio-a85209bcc147c77f0a3e3bb3d40e8d4846c8687217c0e3866d96da5ee20c5116 WatchSource:0}: Error finding container a85209bcc147c77f0a3e3bb3d40e8d4846c8687217c0e3866d96da5ee20c5116: Status 404 returned error can't find the container with id a85209bcc147c77f0a3e3bb3d40e8d4846c8687217c0e3866d96da5ee20c5116 Nov 02 18:16:26.297137 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:26.297120 4471 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 18:16:26.578694 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:26.578610 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67b9f44df6-ncjk9" event={"ID":"57d9041a-7a4a-4179-bcc2-05d21991cbef","Type":"ContainerStarted","Data":"a85209bcc147c77f0a3e3bb3d40e8d4846c8687217c0e3866d96da5ee20c5116"} Nov 02 18:16:28.601071 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:28.601032 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67b9f44df6-ncjk9" event={"ID":"57d9041a-7a4a-4179-bcc2-05d21991cbef","Type":"ContainerStarted","Data":"e24dd5186eca31ea9fe47441c35020fee25af73fba0e81cd2541d803a197ee8c"} Nov 02 18:16:28.601511 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:28.601132 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-67b9f44df6-ncjk9" Nov 02 18:16:28.646546 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:28.646490 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-67b9f44df6-ncjk9" podStartSLOduration=1.9256773360000001 podStartE2EDuration="3.646476274s" totalImagesPullingTime="1.720798938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:16:28.645049928 +0000 UTC m=+2806.340405414" watchObservedRunningTime="2026-11-02 18:16:28.646476274 +0000 UTC m=+2806.341831747" Nov 02 18:16:39.182330 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:16:39.182267 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:16:39.608739 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:39.608708 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-67b9f44df6-ncjk9" Nov 02 18:16:45.244066 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:45.244032 4471 scope.go:122] "RemoveContainer" containerID="32cdac7d1038d4b611349439e522bb34a33f2270bf4dd52f06417698debb4715" Nov 02 18:16:45.254298 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:45.254275 4471 scope.go:122] "RemoveContainer" containerID="0a963746b0cc61037f2ab752432415c0f223c1f491af9aeb2295e6678ff7ee58" Nov 02 18:16:45.262299 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:45.262279 4471 scope.go:122] "RemoveContainer" containerID="52ba382858dfe2119909f86972e5a52f2b1bada1c087726856256cd22fafb54a" Nov 02 18:16:45.270156 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:45.270130 4471 scope.go:122] "RemoveContainer" containerID="ceafb8872240698414d76dea62da1717f0ae739b460a91d9eba0e22f710479e5" Nov 02 18:16:45.277663 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:45.277643 4471 scope.go:122] "RemoveContainer" containerID="1a022d508a380be3b9103abe792e805d2c3f24a8ee8ba1689b731cad78cfb6fa" Nov 02 18:16:47.827650 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.827608 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr"] Nov 02 18:16:47.828124 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.828008 4471 cpu_manager.go:414] "containerMap: removing container" podUID="7d21c3fe-6394-409f-86f1-cae34e9b6ee2" containerName="cleanup" Nov 02 18:16:47.828124 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.828020 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d21c3fe-6394-409f-86f1-cae34e9b6ee2" containerName="cleanup" Nov 02 18:16:47.828124 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.828103 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="684a3084-e591-4e7c-8e0d-7f2d65211a2a" containerName="istio-proxy" Nov 02 18:16:47.828124 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.828114 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d21c3fe-6394-409f-86f1-cae34e9b6ee2" containerName="cleanup" Nov 02 18:16:47.833064 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.833041 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" Nov 02 18:16:47.842736 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.842708 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2"] Nov 02 18:16:47.846651 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.846632 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" Nov 02 18:16:47.854497 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.854462 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-lkf8q\"" Nov 02 18:16:47.855882 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.855857 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr"] Nov 02 18:16:47.868402 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.868372 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3e0f41-024e-4c23-80c4-5cc5b5d08331-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-k97sr\" (UID: \"ef3e0f41-024e-4c23-80c4-5cc5b5d08331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" Nov 02 18:16:47.868544 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.868419 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0a1b02-6f85-4e92-be26-a9f0cb390cef-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-cv4r2\" (UID: \"6c0a1b02-6f85-4e92-be26-a9f0cb390cef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" Nov 02 18:16:47.868544 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.868449 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c0a1b02-6f85-4e92-be26-a9f0cb390cef-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-cv4r2\" (UID: \"6c0a1b02-6f85-4e92-be26-a9f0cb390cef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" Nov 02 18:16:47.868544 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.868483 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0a1b02-6f85-4e92-be26-a9f0cb390cef-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-cv4r2\" (UID: \"6c0a1b02-6f85-4e92-be26-a9f0cb390cef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" Nov 02 18:16:47.868544 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.868511 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef3e0f41-024e-4c23-80c4-5cc5b5d08331-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-k97sr\" (UID: \"ef3e0f41-024e-4c23-80c4-5cc5b5d08331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" Nov 02 18:16:47.868797 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.868617 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c0a1b02-6f85-4e92-be26-a9f0cb390cef-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-cv4r2\" (UID: \"6c0a1b02-6f85-4e92-be26-a9f0cb390cef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" Nov 02 18:16:47.868797 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.868715 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3e0f41-024e-4c23-80c4-5cc5b5d08331-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-k97sr\" (UID: \"ef3e0f41-024e-4c23-80c4-5cc5b5d08331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" Nov 02 18:16:47.868797 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.868751 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef3e0f41-024e-4c23-80c4-5cc5b5d08331-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-k97sr\" (UID: \"ef3e0f41-024e-4c23-80c4-5cc5b5d08331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" Nov 02 18:16:47.868797 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.868786 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c0a1b02-6f85-4e92-be26-a9f0cb390cef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-cv4r2\" (UID: \"6c0a1b02-6f85-4e92-be26-a9f0cb390cef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" Nov 02 18:16:47.869019 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.868813 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef3e0f41-024e-4c23-80c4-5cc5b5d08331-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-k97sr\" (UID: \"ef3e0f41-024e-4c23-80c4-5cc5b5d08331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" Nov 02 18:16:47.869019 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.868851 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef3e0f41-024e-4c23-80c4-5cc5b5d08331-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-k97sr\" (UID: \"ef3e0f41-024e-4c23-80c4-5cc5b5d08331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" Nov 02 18:16:47.869019 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.868881 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef3e0f41-024e-4c23-80c4-5cc5b5d08331-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-k97sr\" (UID: \"ef3e0f41-024e-4c23-80c4-5cc5b5d08331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" Nov 02 18:16:47.869019 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.868911 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c0a1b02-6f85-4e92-be26-a9f0cb390cef-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-cv4r2\" (UID: \"6c0a1b02-6f85-4e92-be26-a9f0cb390cef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" Nov 02 18:16:47.869019 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.868940 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c0a1b02-6f85-4e92-be26-a9f0cb390cef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-cv4r2\" (UID: \"6c0a1b02-6f85-4e92-be26-a9f0cb390cef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" Nov 02 18:16:47.870860 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.869188 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkv5d\" (UniqueName: \"kubernetes.io/projected/6c0a1b02-6f85-4e92-be26-a9f0cb390cef-kube-api-access-kkv5d\") pod \"maas-default-gateway-openshift-default-76f4876667-cv4r2\" (UID: \"6c0a1b02-6f85-4e92-be26-a9f0cb390cef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" Nov 02 18:16:47.870860 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.869225 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c0a1b02-6f85-4e92-be26-a9f0cb390cef-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-cv4r2\" (UID: \"6c0a1b02-6f85-4e92-be26-a9f0cb390cef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" Nov 02 18:16:47.870860 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.869268 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef3e0f41-024e-4c23-80c4-5cc5b5d08331-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-k97sr\" (UID: \"ef3e0f41-024e-4c23-80c4-5cc5b5d08331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" Nov 02 18:16:47.870860 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.869358 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czltr\" (UniqueName: \"kubernetes.io/projected/ef3e0f41-024e-4c23-80c4-5cc5b5d08331-kube-api-access-czltr\") pod \"maas-default-gateway-openshift-default-76f4876667-k97sr\" (UID: \"ef3e0f41-024e-4c23-80c4-5cc5b5d08331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" Nov 02 18:16:47.870860 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.869839 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2"] Nov 02 18:16:47.970583 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.970550 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0a1b02-6f85-4e92-be26-a9f0cb390cef-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-cv4r2\" (UID: \"6c0a1b02-6f85-4e92-be26-a9f0cb390cef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" Nov 02 18:16:47.970787 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.970593 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef3e0f41-024e-4c23-80c4-5cc5b5d08331-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-k97sr\" (UID: \"ef3e0f41-024e-4c23-80c4-5cc5b5d08331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" Nov 02 18:16:47.970787 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.970718 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c0a1b02-6f85-4e92-be26-a9f0cb390cef-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-cv4r2\" (UID: \"6c0a1b02-6f85-4e92-be26-a9f0cb390cef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" Nov 02 18:16:47.970787 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.970770 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3e0f41-024e-4c23-80c4-5cc5b5d08331-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-k97sr\" (UID: \"ef3e0f41-024e-4c23-80c4-5cc5b5d08331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" Nov 02 18:16:47.970984 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.970793 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef3e0f41-024e-4c23-80c4-5cc5b5d08331-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-k97sr\" (UID: \"ef3e0f41-024e-4c23-80c4-5cc5b5d08331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" Nov 02 18:16:47.970984 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.970815 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c0a1b02-6f85-4e92-be26-a9f0cb390cef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-cv4r2\" (UID: \"6c0a1b02-6f85-4e92-be26-a9f0cb390cef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" Nov 02 18:16:47.970984 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.970834 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef3e0f41-024e-4c23-80c4-5cc5b5d08331-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-k97sr\" (UID: \"ef3e0f41-024e-4c23-80c4-5cc5b5d08331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" Nov 02 18:16:47.970984 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.970863 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef3e0f41-024e-4c23-80c4-5cc5b5d08331-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-k97sr\" (UID: \"ef3e0f41-024e-4c23-80c4-5cc5b5d08331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" Nov 02 18:16:47.970984 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.970895 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef3e0f41-024e-4c23-80c4-5cc5b5d08331-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-k97sr\" (UID: \"ef3e0f41-024e-4c23-80c4-5cc5b5d08331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" Nov 02 18:16:47.970984 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.970926 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c0a1b02-6f85-4e92-be26-a9f0cb390cef-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-cv4r2\" (UID: \"6c0a1b02-6f85-4e92-be26-a9f0cb390cef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" Nov 02 18:16:47.970984 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.970957 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c0a1b02-6f85-4e92-be26-a9f0cb390cef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-cv4r2\" (UID: \"6c0a1b02-6f85-4e92-be26-a9f0cb390cef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" Nov 02 18:16:47.971406 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.970984 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0a1b02-6f85-4e92-be26-a9f0cb390cef-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-cv4r2\" (UID: \"6c0a1b02-6f85-4e92-be26-a9f0cb390cef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" Nov 02 18:16:47.971406 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.971010 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkv5d\" (UniqueName: \"kubernetes.io/projected/6c0a1b02-6f85-4e92-be26-a9f0cb390cef-kube-api-access-kkv5d\") pod \"maas-default-gateway-openshift-default-76f4876667-cv4r2\" (UID: \"6c0a1b02-6f85-4e92-be26-a9f0cb390cef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" Nov 02 18:16:47.971406 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.971040 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c0a1b02-6f85-4e92-be26-a9f0cb390cef-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-cv4r2\" (UID: \"6c0a1b02-6f85-4e92-be26-a9f0cb390cef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" Nov 02 18:16:47.971406 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.971060 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c0a1b02-6f85-4e92-be26-a9f0cb390cef-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-cv4r2\" (UID: \"6c0a1b02-6f85-4e92-be26-a9f0cb390cef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" Nov 02 18:16:47.971406 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.971081 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef3e0f41-024e-4c23-80c4-5cc5b5d08331-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-k97sr\" (UID: \"ef3e0f41-024e-4c23-80c4-5cc5b5d08331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" Nov 02 18:16:47.971406 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.971117 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czltr\" (UniqueName: \"kubernetes.io/projected/ef3e0f41-024e-4c23-80c4-5cc5b5d08331-kube-api-access-czltr\") pod \"maas-default-gateway-openshift-default-76f4876667-k97sr\" (UID: \"ef3e0f41-024e-4c23-80c4-5cc5b5d08331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" Nov 02 18:16:47.971406 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.971177 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3e0f41-024e-4c23-80c4-5cc5b5d08331-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-k97sr\" (UID: \"ef3e0f41-024e-4c23-80c4-5cc5b5d08331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" Nov 02 18:16:47.971406 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.971194 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3e0f41-024e-4c23-80c4-5cc5b5d08331-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-k97sr\" (UID: \"ef3e0f41-024e-4c23-80c4-5cc5b5d08331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" Nov 02 18:16:47.971406 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.971206 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0a1b02-6f85-4e92-be26-a9f0cb390cef-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-cv4r2\" (UID: \"6c0a1b02-6f85-4e92-be26-a9f0cb390cef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" Nov 02 18:16:47.971406 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.971251 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c0a1b02-6f85-4e92-be26-a9f0cb390cef-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-cv4r2\" (UID: \"6c0a1b02-6f85-4e92-be26-a9f0cb390cef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" Nov 02 18:16:47.971406 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.971263 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c0a1b02-6f85-4e92-be26-a9f0cb390cef-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-cv4r2\" (UID: \"6c0a1b02-6f85-4e92-be26-a9f0cb390cef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" Nov 02 18:16:47.971994 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.971475 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0a1b02-6f85-4e92-be26-a9f0cb390cef-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-cv4r2\" (UID: \"6c0a1b02-6f85-4e92-be26-a9f0cb390cef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" Nov 02 18:16:47.971994 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.971693 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef3e0f41-024e-4c23-80c4-5cc5b5d08331-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-k97sr\" (UID: \"ef3e0f41-024e-4c23-80c4-5cc5b5d08331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" Nov 02 18:16:47.971994 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.971777 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c0a1b02-6f85-4e92-be26-a9f0cb390cef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-cv4r2\" (UID: \"6c0a1b02-6f85-4e92-be26-a9f0cb390cef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" Nov 02 18:16:47.972177 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.972145 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef3e0f41-024e-4c23-80c4-5cc5b5d08331-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-k97sr\" (UID: \"ef3e0f41-024e-4c23-80c4-5cc5b5d08331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" Nov 02 18:16:47.972282 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.972242 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3e0f41-024e-4c23-80c4-5cc5b5d08331-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-k97sr\" (UID: \"ef3e0f41-024e-4c23-80c4-5cc5b5d08331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" Nov 02 18:16:47.972480 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.972460 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef3e0f41-024e-4c23-80c4-5cc5b5d08331-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-k97sr\" (UID: \"ef3e0f41-024e-4c23-80c4-5cc5b5d08331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" Nov 02 18:16:47.974132 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.974108 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c0a1b02-6f85-4e92-be26-a9f0cb390cef-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-cv4r2\" (UID: \"6c0a1b02-6f85-4e92-be26-a9f0cb390cef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" Nov 02 18:16:47.974243 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.974148 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef3e0f41-024e-4c23-80c4-5cc5b5d08331-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-k97sr\" (UID: \"ef3e0f41-024e-4c23-80c4-5cc5b5d08331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" Nov 02 18:16:47.974367 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.974344 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c0a1b02-6f85-4e92-be26-a9f0cb390cef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-cv4r2\" (UID: \"6c0a1b02-6f85-4e92-be26-a9f0cb390cef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" Nov 02 18:16:47.974512 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.974495 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef3e0f41-024e-4c23-80c4-5cc5b5d08331-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-k97sr\" (UID: \"ef3e0f41-024e-4c23-80c4-5cc5b5d08331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" Nov 02 18:16:47.994892 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.994860 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef3e0f41-024e-4c23-80c4-5cc5b5d08331-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-k97sr\" (UID: \"ef3e0f41-024e-4c23-80c4-5cc5b5d08331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" Nov 02 18:16:47.995077 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.995057 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c0a1b02-6f85-4e92-be26-a9f0cb390cef-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-cv4r2\" (UID: \"6c0a1b02-6f85-4e92-be26-a9f0cb390cef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" Nov 02 18:16:47.995599 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.995583 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czltr\" (UniqueName: \"kubernetes.io/projected/ef3e0f41-024e-4c23-80c4-5cc5b5d08331-kube-api-access-czltr\") pod \"maas-default-gateway-openshift-default-76f4876667-k97sr\" (UID: \"ef3e0f41-024e-4c23-80c4-5cc5b5d08331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" Nov 02 18:16:47.995792 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:47.995775 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkv5d\" (UniqueName: \"kubernetes.io/projected/6c0a1b02-6f85-4e92-be26-a9f0cb390cef-kube-api-access-kkv5d\") pod \"maas-default-gateway-openshift-default-76f4876667-cv4r2\" (UID: \"6c0a1b02-6f85-4e92-be26-a9f0cb390cef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" Nov 02 18:16:48.147803 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:48.147724 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" Nov 02 18:16:48.160861 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:48.160836 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" Nov 02 18:16:48.463957 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:16:48.463926 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef3e0f41_024e_4c23_80c4_5cc5b5d08331.slice/crio-ab69977425df0f1aa19ca2ad13a4112e51545b020450f4a98de1b0480679cef2 WatchSource:0}: Error finding container ab69977425df0f1aa19ca2ad13a4112e51545b020450f4a98de1b0480679cef2: Status 404 returned error can't find the container with id ab69977425df0f1aa19ca2ad13a4112e51545b020450f4a98de1b0480679cef2 Nov 02 18:16:48.466213 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:48.466178 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:16:48.466363 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:48.466261 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:16:48.466363 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:48.466348 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:16:48.475536 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:16:48.475502 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c0a1b02_6f85_4e92_be26_a9f0cb390cef.slice/crio-775578cc7bce5dad71e1206ba39a6bf0d87da1518073082c3e836b51ee9d530a WatchSource:0}: Error finding container 775578cc7bce5dad71e1206ba39a6bf0d87da1518073082c3e836b51ee9d530a: Status 404 returned error can't find the container with id 775578cc7bce5dad71e1206ba39a6bf0d87da1518073082c3e836b51ee9d530a Nov 02 18:16:48.476417 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:48.476392 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr"] Nov 02 18:16:48.478100 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:48.478080 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2"] Nov 02 18:16:48.478152 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:48.478110 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:16:48.478189 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:48.478165 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:16:48.478221 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:48.478201 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:16:48.694126 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:48.694090 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" event={"ID":"6c0a1b02-6f85-4e92-be26-a9f0cb390cef","Type":"ContainerStarted","Data":"d8522bc13b5f27b45ea883a9faf88635f4ea68aaa24b58c801c00fa6e054690b"} Nov 02 18:16:48.694342 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:48.694156 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" event={"ID":"6c0a1b02-6f85-4e92-be26-a9f0cb390cef","Type":"ContainerStarted","Data":"775578cc7bce5dad71e1206ba39a6bf0d87da1518073082c3e836b51ee9d530a"} Nov 02 18:16:48.695909 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:48.695881 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" event={"ID":"ef3e0f41-024e-4c23-80c4-5cc5b5d08331","Type":"ContainerStarted","Data":"5458c3d734c7374357de1c51a6bcd550b1d42b668caad428c71e17f503313e25"} Nov 02 18:16:48.696020 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:48.695933 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" event={"ID":"ef3e0f41-024e-4c23-80c4-5cc5b5d08331","Type":"ContainerStarted","Data":"ab69977425df0f1aa19ca2ad13a4112e51545b020450f4a98de1b0480679cef2"} Nov 02 18:16:48.742055 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:48.741941 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" podStartSLOduration=1.741923339 podStartE2EDuration="1.741923339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:16:48.739453134 +0000 UTC m=+2826.434808640" watchObservedRunningTime="2026-11-02 18:16:48.741923339 +0000 UTC m=+2826.437278813" Nov 02 18:16:48.786419 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:48.786370 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" podStartSLOduration=1.786355181 podStartE2EDuration="1.786355181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:16:48.784699187 +0000 UTC m=+2826.480054661" watchObservedRunningTime="2026-11-02 18:16:48.786355181 +0000 UTC m=+2826.481710653" Nov 02 18:16:49.148484 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:49.148430 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" Nov 02 18:16:49.149869 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:49.149843 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-k97sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 02 18:16:49.149949 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:49.149889 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" podUID="ef3e0f41-024e-4c23-80c4-5cc5b5d08331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 02 18:16:49.169331 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:49.164944 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" Nov 02 18:16:49.169331 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:49.165101 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-cv4r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 02 18:16:49.169331 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:49.165151 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" podUID="6c0a1b02-6f85-4e92-be26-a9f0cb390cef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 02 18:16:50.148694 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:50.148660 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-k97sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 02 18:16:50.149119 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:50.148711 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" podUID="ef3e0f41-024e-4c23-80c4-5cc5b5d08331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 02 18:16:50.163741 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:50.163702 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-cv4r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 02 18:16:50.164031 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:50.164000 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" podUID="6c0a1b02-6f85-4e92-be26-a9f0cb390cef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 02 18:16:51.148374 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:51.148335 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-k97sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 02 18:16:51.148634 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:51.148608 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" podUID="ef3e0f41-024e-4c23-80c4-5cc5b5d08331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 02 18:16:51.166330 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:51.163722 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-cv4r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 02 18:16:51.166330 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:51.163785 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" podUID="6c0a1b02-6f85-4e92-be26-a9f0cb390cef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 02 18:16:52.148093 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:52.148061 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-k97sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 02 18:16:52.148380 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:52.148354 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" podUID="ef3e0f41-024e-4c23-80c4-5cc5b5d08331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 02 18:16:52.167331 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:52.163661 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-cv4r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 02 18:16:52.167331 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:52.163717 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" podUID="6c0a1b02-6f85-4e92-be26-a9f0cb390cef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 02 18:16:53.148110 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:53.148076 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-k97sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 02 18:16:53.148411 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:53.148379 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" podUID="ef3e0f41-024e-4c23-80c4-5cc5b5d08331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 02 18:16:53.173724 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:53.173687 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-cv4r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 02 18:16:53.174350 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:53.174293 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" podUID="6c0a1b02-6f85-4e92-be26-a9f0cb390cef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 02 18:16:54.148586 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:54.148554 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-k97sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 02 18:16:54.148781 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:54.148607 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" podUID="ef3e0f41-024e-4c23-80c4-5cc5b5d08331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 02 18:16:54.168340 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:54.163665 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-cv4r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 02 18:16:54.168340 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:54.163733 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" podUID="6c0a1b02-6f85-4e92-be26-a9f0cb390cef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 02 18:16:55.148415 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:55.148285 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-k97sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 02 18:16:55.148415 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:55.148365 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" podUID="ef3e0f41-024e-4c23-80c4-5cc5b5d08331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 02 18:16:55.161622 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:55.161587 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-cv4r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 02 18:16:55.161771 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:55.161633 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" podUID="6c0a1b02-6f85-4e92-be26-a9f0cb390cef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 02 18:16:57.148983 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:57.148928 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-k97sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:16:57.149545 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:57.149033 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" podUID="ef3e0f41-024e-4c23-80c4-5cc5b5d08331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:16:57.162058 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:57.162015 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-cv4r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:16:57.162340 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:57.162103 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" podUID="6c0a1b02-6f85-4e92-be26-a9f0cb390cef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:16:58.190521 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:58.190470 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-k97sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:16:58.191028 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:58.190554 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" podUID="ef3e0f41-024e-4c23-80c4-5cc5b5d08331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:16:58.204586 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:58.204539 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-cv4r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:16:58.204751 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:58.204600 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" podUID="6c0a1b02-6f85-4e92-be26-a9f0cb390cef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:16:58.754139 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:58.754099 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-67b9f44df6-ncjk9"] Nov 02 18:16:58.754455 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:58.754366 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-67b9f44df6-ncjk9" podUID="57d9041a-7a4a-4179-bcc2-05d21991cbef" containerName="manager" containerID="cri-o://e24dd5186eca31ea9fe47441c35020fee25af73fba0e81cd2541d803a197ee8c" gracePeriod=30 Nov 02 18:16:59.057781 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:59.057518 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67b9f44df6-ncjk9" Nov 02 18:16:59.144975 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:59.144928 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57d9041a-7a4a-4179-bcc2-05d21991cbef-webhook-cert\" (UniqueName: \"kubernetes.io/secret/57d9041a-7a4a-4179-bcc2-05d21991cbef-webhook-cert\") pod \"57d9041a-7a4a-4179-bcc2-05d21991cbef\" (UID: \"57d9041a-7a4a-4179-bcc2-05d21991cbef\") " Nov 02 18:16:59.145174 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:59.144990 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57d9041a-7a4a-4179-bcc2-05d21991cbef-kube-api-access-xl2hf\" (UniqueName: \"kubernetes.io/projected/57d9041a-7a4a-4179-bcc2-05d21991cbef-kube-api-access-xl2hf\") pod \"57d9041a-7a4a-4179-bcc2-05d21991cbef\" (UID: \"57d9041a-7a4a-4179-bcc2-05d21991cbef\") " Nov 02 18:16:59.145174 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:59.145031 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57d9041a-7a4a-4179-bcc2-05d21991cbef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/57d9041a-7a4a-4179-bcc2-05d21991cbef-metrics-certs\") pod \"57d9041a-7a4a-4179-bcc2-05d21991cbef\" (UID: \"57d9041a-7a4a-4179-bcc2-05d21991cbef\") " Nov 02 18:16:59.147742 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:59.147711 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57d9041a-7a4a-4179-bcc2-05d21991cbef-kube-api-access-xl2hf" pod "57d9041a-7a4a-4179-bcc2-05d21991cbef" (UID: "57d9041a-7a4a-4179-bcc2-05d21991cbef"). InnerVolumeSpecName "kube-api-access-xl2hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:16:59.147861 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:59.147759 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57d9041a-7a4a-4179-bcc2-05d21991cbef-webhook-cert" pod "57d9041a-7a4a-4179-bcc2-05d21991cbef" (UID: "57d9041a-7a4a-4179-bcc2-05d21991cbef"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:16:59.148066 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:59.148045 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57d9041a-7a4a-4179-bcc2-05d21991cbef-metrics-certs" pod "57d9041a-7a4a-4179-bcc2-05d21991cbef" (UID: "57d9041a-7a4a-4179-bcc2-05d21991cbef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:16:59.205690 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:59.205648 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-cv4r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:16:59.206170 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:59.205729 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" podUID="6c0a1b02-6f85-4e92-be26-a9f0cb390cef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:16:59.232483 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:59.232441 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-k97sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:16:59.232629 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:59.232508 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" podUID="ef3e0f41-024e-4c23-80c4-5cc5b5d08331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:16:59.246861 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:59.246834 4471 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57d9041a-7a4a-4179-bcc2-05d21991cbef-webhook-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:16:59.246861 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:59.246860 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl2hf\" (UniqueName: \"kubernetes.io/projected/57d9041a-7a4a-4179-bcc2-05d21991cbef-kube-api-access-xl2hf\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:16:59.247033 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:59.246870 4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57d9041a-7a4a-4179-bcc2-05d21991cbef-metrics-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:16:59.329683 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:59.329594 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" Nov 02 18:16:59.330019 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:59.329994 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" Nov 02 18:16:59.330958 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:59.330940 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" Nov 02 18:16:59.394522 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:59.394491 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" Nov 02 18:16:59.394828 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:59.394798 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" Nov 02 18:16:59.395545 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:59.395526 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" Nov 02 18:16:59.767280 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:59.767246 4471 generic.go:356] "Generic (PLEG): container finished" podID="57d9041a-7a4a-4179-bcc2-05d21991cbef" containerID="e24dd5186eca31ea9fe47441c35020fee25af73fba0e81cd2541d803a197ee8c" exitCode=0 Nov 02 18:16:59.767462 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:59.767294 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67b9f44df6-ncjk9" Nov 02 18:16:59.767462 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:59.767333 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67b9f44df6-ncjk9" event={"ID":"57d9041a-7a4a-4179-bcc2-05d21991cbef","Type":"ContainerDied","Data":"e24dd5186eca31ea9fe47441c35020fee25af73fba0e81cd2541d803a197ee8c"} Nov 02 18:16:59.767462 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:59.767387 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67b9f44df6-ncjk9" event={"ID":"57d9041a-7a4a-4179-bcc2-05d21991cbef","Type":"ContainerDied","Data":"a85209bcc147c77f0a3e3bb3d40e8d4846c8687217c0e3866d96da5ee20c5116"} Nov 02 18:16:59.767462 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:59.767411 4471 scope.go:122] "RemoveContainer" containerID="e24dd5186eca31ea9fe47441c35020fee25af73fba0e81cd2541d803a197ee8c" Nov 02 18:16:59.776504 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:59.776488 4471 scope.go:122] "RemoveContainer" containerID="e24dd5186eca31ea9fe47441c35020fee25af73fba0e81cd2541d803a197ee8c" Nov 02 18:16:59.776797 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:16:59.776776 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e24dd5186eca31ea9fe47441c35020fee25af73fba0e81cd2541d803a197ee8c\": container with ID starting with e24dd5186eca31ea9fe47441c35020fee25af73fba0e81cd2541d803a197ee8c not found: ID does not exist" containerID="e24dd5186eca31ea9fe47441c35020fee25af73fba0e81cd2541d803a197ee8c" Nov 02 18:16:59.776871 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:59.776803 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e24dd5186eca31ea9fe47441c35020fee25af73fba0e81cd2541d803a197ee8c"} err="failed to get container status \"e24dd5186eca31ea9fe47441c35020fee25af73fba0e81cd2541d803a197ee8c\": rpc error: code = NotFound desc = could not find container \"e24dd5186eca31ea9fe47441c35020fee25af73fba0e81cd2541d803a197ee8c\": container with ID starting with e24dd5186eca31ea9fe47441c35020fee25af73fba0e81cd2541d803a197ee8c not found: ID does not exist" Nov 02 18:16:59.819417 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:59.819391 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-67b9f44df6-ncjk9"] Nov 02 18:16:59.832102 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:16:59.832075 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-67b9f44df6-ncjk9"] Nov 02 18:17:00.439722 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:00.439686 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57d9041a-7a4a-4179-bcc2-05d21991cbef" path="/var/lib/kubelet/pods/57d9041a-7a4a-4179-bcc2-05d21991cbef/volumes" Nov 02 18:17:01.024504 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:01.024463 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-67b9f44df6-zft8x"] Nov 02 18:17:01.025509 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:01.025486 4471 cpu_manager.go:414] "containerMap: removing container" podUID="57d9041a-7a4a-4179-bcc2-05d21991cbef" containerName="manager" Nov 02 18:17:01.025643 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:01.025510 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57d9041a-7a4a-4179-bcc2-05d21991cbef" containerName="manager" Nov 02 18:17:01.025744 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:01.025731 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="57d9041a-7a4a-4179-bcc2-05d21991cbef" containerName="manager" Nov 02 18:17:01.029852 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:01.029821 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67b9f44df6-zft8x" Nov 02 18:17:01.057280 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:01.057241 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kv77t\"" Nov 02 18:17:01.058521 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:01.058491 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67b9f44df6-zft8x"] Nov 02 18:17:01.069434 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:01.069398 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 18:17:01.069574 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:01.069515 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 18:17:01.069949 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:01.069931 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 18:17:01.167613 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:01.167578 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sqg6\" (UniqueName: \"kubernetes.io/projected/eae8886c-0696-40df-b91f-2fbd5b50e65e-kube-api-access-4sqg6\") pod \"maas-controller-67b9f44df6-zft8x\" (UID: \"eae8886c-0696-40df-b91f-2fbd5b50e65e\") " pod="opendatahub/maas-controller-67b9f44df6-zft8x" Nov 02 18:17:01.167776 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:01.167625 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eae8886c-0696-40df-b91f-2fbd5b50e65e-metrics-certs\") pod \"maas-controller-67b9f44df6-zft8x\" (UID: \"eae8886c-0696-40df-b91f-2fbd5b50e65e\") " pod="opendatahub/maas-controller-67b9f44df6-zft8x" Nov 02 18:17:01.167776 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:01.167644 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eae8886c-0696-40df-b91f-2fbd5b50e65e-webhook-cert\") pod \"maas-controller-67b9f44df6-zft8x\" (UID: \"eae8886c-0696-40df-b91f-2fbd5b50e65e\") " pod="opendatahub/maas-controller-67b9f44df6-zft8x" Nov 02 18:17:01.268242 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:01.268207 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sqg6\" (UniqueName: \"kubernetes.io/projected/eae8886c-0696-40df-b91f-2fbd5b50e65e-kube-api-access-4sqg6\") pod \"maas-controller-67b9f44df6-zft8x\" (UID: \"eae8886c-0696-40df-b91f-2fbd5b50e65e\") " pod="opendatahub/maas-controller-67b9f44df6-zft8x" Nov 02 18:17:01.268457 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:01.268251 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eae8886c-0696-40df-b91f-2fbd5b50e65e-metrics-certs\") pod \"maas-controller-67b9f44df6-zft8x\" (UID: \"eae8886c-0696-40df-b91f-2fbd5b50e65e\") " pod="opendatahub/maas-controller-67b9f44df6-zft8x" Nov 02 18:17:01.268457 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:01.268271 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eae8886c-0696-40df-b91f-2fbd5b50e65e-webhook-cert\") pod \"maas-controller-67b9f44df6-zft8x\" (UID: \"eae8886c-0696-40df-b91f-2fbd5b50e65e\") " pod="opendatahub/maas-controller-67b9f44df6-zft8x" Nov 02 18:17:01.270745 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:01.270722 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eae8886c-0696-40df-b91f-2fbd5b50e65e-webhook-cert\") pod \"maas-controller-67b9f44df6-zft8x\" (UID: \"eae8886c-0696-40df-b91f-2fbd5b50e65e\") " pod="opendatahub/maas-controller-67b9f44df6-zft8x" Nov 02 18:17:01.270854 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:01.270825 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eae8886c-0696-40df-b91f-2fbd5b50e65e-metrics-certs\") pod \"maas-controller-67b9f44df6-zft8x\" (UID: \"eae8886c-0696-40df-b91f-2fbd5b50e65e\") " pod="opendatahub/maas-controller-67b9f44df6-zft8x" Nov 02 18:17:01.292805 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:01.292742 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sqg6\" (UniqueName: \"kubernetes.io/projected/eae8886c-0696-40df-b91f-2fbd5b50e65e-kube-api-access-4sqg6\") pod \"maas-controller-67b9f44df6-zft8x\" (UID: \"eae8886c-0696-40df-b91f-2fbd5b50e65e\") " pod="opendatahub/maas-controller-67b9f44df6-zft8x" Nov 02 18:17:01.343202 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:01.343176 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67b9f44df6-zft8x" Nov 02 18:17:01.669841 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:01.669810 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67b9f44df6-zft8x"] Nov 02 18:17:01.671619 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:17:01.671591 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae8886c_0696_40df_b91f_2fbd5b50e65e.slice/crio-c5a6d820584de31d0c8c081dfdfd86d301afc1d620c29d831dea07792fac4a36 WatchSource:0}: Error finding container c5a6d820584de31d0c8c081dfdfd86d301afc1d620c29d831dea07792fac4a36: Status 404 returned error can't find the container with id c5a6d820584de31d0c8c081dfdfd86d301afc1d620c29d831dea07792fac4a36 Nov 02 18:17:01.778126 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:01.778085 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67b9f44df6-zft8x" event={"ID":"eae8886c-0696-40df-b91f-2fbd5b50e65e","Type":"ContainerStarted","Data":"c5a6d820584de31d0c8c081dfdfd86d301afc1d620c29d831dea07792fac4a36"} Nov 02 18:17:02.784536 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:02.784499 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67b9f44df6-zft8x" event={"ID":"eae8886c-0696-40df-b91f-2fbd5b50e65e","Type":"ContainerStarted","Data":"6c63d6603e877df314e012e4192ccbc62f4539584d5da92037f83db7111dba13"} Nov 02 18:17:02.784964 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:02.784578 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-67b9f44df6-zft8x" Nov 02 18:17:02.829429 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:02.829367 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-67b9f44df6-zft8x" podStartSLOduration=2.534894392 podStartE2EDuration="2.82935185s" totalImagesPullingTime="294.457458ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:17:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:17:02.827988412 +0000 UTC m=+2840.523343887" watchObservedRunningTime="2026-11-02 18:17:02.82935185 +0000 UTC m=+2840.524707322" Nov 02 18:17:02.873791 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:02.873750 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn"] Nov 02 18:17:02.876709 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:02.876685 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" Nov 02 18:17:02.902401 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:02.902370 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn"] Nov 02 18:17:02.906675 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:02.906647 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2"] Nov 02 18:17:02.909561 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:02.909540 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" Nov 02 18:17:02.933939 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:02.933902 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2"] Nov 02 18:17:02.984461 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:02.984425 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a482b42d-8782-433f-a05f-391d4516a850-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-5hhwn\" (UID: \"a482b42d-8782-433f-a05f-391d4516a850\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" Nov 02 18:17:02.984649 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:02.984481 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a482b42d-8782-433f-a05f-391d4516a850-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5hhwn\" (UID: \"a482b42d-8782-433f-a05f-391d4516a850\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" Nov 02 18:17:02.984649 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:02.984608 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a482b42d-8782-433f-a05f-391d4516a850-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-5hhwn\" (UID: \"a482b42d-8782-433f-a05f-391d4516a850\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" Nov 02 18:17:02.984745 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:02.984651 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c40d5295-d354-4be9-89b8-6bd96ae2e056-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-f6kf2\" (UID: \"c40d5295-d354-4be9-89b8-6bd96ae2e056\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" Nov 02 18:17:02.984745 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:02.984673 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6mvw\" (UniqueName: \"kubernetes.io/projected/c40d5295-d354-4be9-89b8-6bd96ae2e056-kube-api-access-v6mvw\") pod \"maas-default-gateway-openshift-default-76f4876667-f6kf2\" (UID: \"c40d5295-d354-4be9-89b8-6bd96ae2e056\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" Nov 02 18:17:02.984745 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:02.984733 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c40d5295-d354-4be9-89b8-6bd96ae2e056-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-f6kf2\" (UID: \"c40d5295-d354-4be9-89b8-6bd96ae2e056\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" Nov 02 18:17:02.984874 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:02.984804 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40d5295-d354-4be9-89b8-6bd96ae2e056-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-f6kf2\" (UID: \"c40d5295-d354-4be9-89b8-6bd96ae2e056\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" Nov 02 18:17:02.984874 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:02.984842 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40d5295-d354-4be9-89b8-6bd96ae2e056-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-f6kf2\" (UID: \"c40d5295-d354-4be9-89b8-6bd96ae2e056\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" Nov 02 18:17:02.984961 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:02.984876 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c40d5295-d354-4be9-89b8-6bd96ae2e056-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-f6kf2\" (UID: \"c40d5295-d354-4be9-89b8-6bd96ae2e056\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" Nov 02 18:17:02.985016 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:02.984948 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcx7f\" (UniqueName: \"kubernetes.io/projected/a482b42d-8782-433f-a05f-391d4516a850-kube-api-access-dcx7f\") pod \"maas-default-gateway-openshift-default-76f4876667-5hhwn\" (UID: \"a482b42d-8782-433f-a05f-391d4516a850\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" Nov 02 18:17:02.985016 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:02.985001 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c40d5295-d354-4be9-89b8-6bd96ae2e056-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-f6kf2\" (UID: \"c40d5295-d354-4be9-89b8-6bd96ae2e056\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" Nov 02 18:17:02.985107 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:02.985033 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a482b42d-8782-433f-a05f-391d4516a850-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5hhwn\" (UID: \"a482b42d-8782-433f-a05f-391d4516a850\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" Nov 02 18:17:02.985107 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:02.985081 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a482b42d-8782-433f-a05f-391d4516a850-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-5hhwn\" (UID: \"a482b42d-8782-433f-a05f-391d4516a850\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" Nov 02 18:17:02.985184 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:02.985119 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c40d5295-d354-4be9-89b8-6bd96ae2e056-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-f6kf2\" (UID: \"c40d5295-d354-4be9-89b8-6bd96ae2e056\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" Nov 02 18:17:02.985220 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:02.985183 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c40d5295-d354-4be9-89b8-6bd96ae2e056-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-f6kf2\" (UID: \"c40d5295-d354-4be9-89b8-6bd96ae2e056\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" Nov 02 18:17:02.985220 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:02.985213 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a482b42d-8782-433f-a05f-391d4516a850-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-5hhwn\" (UID: \"a482b42d-8782-433f-a05f-391d4516a850\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" Nov 02 18:17:02.985298 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:02.985234 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a482b42d-8782-433f-a05f-391d4516a850-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-5hhwn\" (UID: \"a482b42d-8782-433f-a05f-391d4516a850\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" Nov 02 18:17:02.985420 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:02.985403 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a482b42d-8782-433f-a05f-391d4516a850-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-5hhwn\" (UID: \"a482b42d-8782-433f-a05f-391d4516a850\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" Nov 02 18:17:03.085967 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.085892 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcx7f\" (UniqueName: \"kubernetes.io/projected/a482b42d-8782-433f-a05f-391d4516a850-kube-api-access-dcx7f\") pod \"maas-default-gateway-openshift-default-76f4876667-5hhwn\" (UID: \"a482b42d-8782-433f-a05f-391d4516a850\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" Nov 02 18:17:03.085967 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.085933 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c40d5295-d354-4be9-89b8-6bd96ae2e056-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-f6kf2\" (UID: \"c40d5295-d354-4be9-89b8-6bd96ae2e056\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" Nov 02 18:17:03.086183 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.086164 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a482b42d-8782-433f-a05f-391d4516a850-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5hhwn\" (UID: \"a482b42d-8782-433f-a05f-391d4516a850\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" Nov 02 18:17:03.086228 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.086193 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a482b42d-8782-433f-a05f-391d4516a850-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-5hhwn\" (UID: \"a482b42d-8782-433f-a05f-391d4516a850\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" Nov 02 18:17:03.086228 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.086210 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c40d5295-d354-4be9-89b8-6bd96ae2e056-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-f6kf2\" (UID: \"c40d5295-d354-4be9-89b8-6bd96ae2e056\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" Nov 02 18:17:03.086360 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.086245 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c40d5295-d354-4be9-89b8-6bd96ae2e056-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-f6kf2\" (UID: \"c40d5295-d354-4be9-89b8-6bd96ae2e056\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" Nov 02 18:17:03.086360 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.086289 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a482b42d-8782-433f-a05f-391d4516a850-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-5hhwn\" (UID: \"a482b42d-8782-433f-a05f-391d4516a850\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" Nov 02 18:17:03.086360 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.086345 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a482b42d-8782-433f-a05f-391d4516a850-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-5hhwn\" (UID: \"a482b42d-8782-433f-a05f-391d4516a850\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" Nov 02 18:17:03.086527 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.086427 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a482b42d-8782-433f-a05f-391d4516a850-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-5hhwn\" (UID: \"a482b42d-8782-433f-a05f-391d4516a850\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" Nov 02 18:17:03.086527 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.086458 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a482b42d-8782-433f-a05f-391d4516a850-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-5hhwn\" (UID: \"a482b42d-8782-433f-a05f-391d4516a850\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" Nov 02 18:17:03.086662 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.086632 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a482b42d-8782-433f-a05f-391d4516a850-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5hhwn\" (UID: \"a482b42d-8782-433f-a05f-391d4516a850\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" Nov 02 18:17:03.086723 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.086676 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a482b42d-8782-433f-a05f-391d4516a850-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-5hhwn\" (UID: \"a482b42d-8782-433f-a05f-391d4516a850\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" Nov 02 18:17:03.086723 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.086688 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a482b42d-8782-433f-a05f-391d4516a850-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5hhwn\" (UID: \"a482b42d-8782-433f-a05f-391d4516a850\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" Nov 02 18:17:03.086840 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.086718 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c40d5295-d354-4be9-89b8-6bd96ae2e056-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-f6kf2\" (UID: \"c40d5295-d354-4be9-89b8-6bd96ae2e056\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" Nov 02 18:17:03.086840 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.086746 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a482b42d-8782-433f-a05f-391d4516a850-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-5hhwn\" (UID: \"a482b42d-8782-433f-a05f-391d4516a850\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" Nov 02 18:17:03.086840 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.086772 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c40d5295-d354-4be9-89b8-6bd96ae2e056-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-f6kf2\" (UID: \"c40d5295-d354-4be9-89b8-6bd96ae2e056\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" Nov 02 18:17:03.086840 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.086791 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6mvw\" (UniqueName: \"kubernetes.io/projected/c40d5295-d354-4be9-89b8-6bd96ae2e056-kube-api-access-v6mvw\") pod \"maas-default-gateway-openshift-default-76f4876667-f6kf2\" (UID: \"c40d5295-d354-4be9-89b8-6bd96ae2e056\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" Nov 02 18:17:03.086840 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.086813 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c40d5295-d354-4be9-89b8-6bd96ae2e056-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-f6kf2\" (UID: \"c40d5295-d354-4be9-89b8-6bd96ae2e056\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" Nov 02 18:17:03.087124 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.086844 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40d5295-d354-4be9-89b8-6bd96ae2e056-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-f6kf2\" (UID: \"c40d5295-d354-4be9-89b8-6bd96ae2e056\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" Nov 02 18:17:03.087124 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.086871 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40d5295-d354-4be9-89b8-6bd96ae2e056-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-f6kf2\" (UID: \"c40d5295-d354-4be9-89b8-6bd96ae2e056\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" Nov 02 18:17:03.087124 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.086917 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c40d5295-d354-4be9-89b8-6bd96ae2e056-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-f6kf2\" (UID: \"c40d5295-d354-4be9-89b8-6bd96ae2e056\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" Nov 02 18:17:03.087299 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.087127 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c40d5295-d354-4be9-89b8-6bd96ae2e056-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-f6kf2\" (UID: \"c40d5295-d354-4be9-89b8-6bd96ae2e056\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" Nov 02 18:17:03.087404 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.087336 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40d5295-d354-4be9-89b8-6bd96ae2e056-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-f6kf2\" (UID: \"c40d5295-d354-4be9-89b8-6bd96ae2e056\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" Nov 02 18:17:03.087404 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.087378 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c40d5295-d354-4be9-89b8-6bd96ae2e056-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-f6kf2\" (UID: \"c40d5295-d354-4be9-89b8-6bd96ae2e056\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" Nov 02 18:17:03.087535 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.087382 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40d5295-d354-4be9-89b8-6bd96ae2e056-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-f6kf2\" (UID: \"c40d5295-d354-4be9-89b8-6bd96ae2e056\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" Nov 02 18:17:03.087620 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.087598 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a482b42d-8782-433f-a05f-391d4516a850-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5hhwn\" (UID: \"a482b42d-8782-433f-a05f-391d4516a850\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" Nov 02 18:17:03.087730 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.087701 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a482b42d-8782-433f-a05f-391d4516a850-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-5hhwn\" (UID: \"a482b42d-8782-433f-a05f-391d4516a850\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" Nov 02 18:17:03.087939 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.087916 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a482b42d-8782-433f-a05f-391d4516a850-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-5hhwn\" (UID: \"a482b42d-8782-433f-a05f-391d4516a850\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" Nov 02 18:17:03.089286 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.089266 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a482b42d-8782-433f-a05f-391d4516a850-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-5hhwn\" (UID: \"a482b42d-8782-433f-a05f-391d4516a850\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" Nov 02 18:17:03.089286 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.089285 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c40d5295-d354-4be9-89b8-6bd96ae2e056-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-f6kf2\" (UID: \"c40d5295-d354-4be9-89b8-6bd96ae2e056\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" Nov 02 18:17:03.089497 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.089479 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c40d5295-d354-4be9-89b8-6bd96ae2e056-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-f6kf2\" (UID: \"c40d5295-d354-4be9-89b8-6bd96ae2e056\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" Nov 02 18:17:03.089675 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.089659 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a482b42d-8782-433f-a05f-391d4516a850-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-5hhwn\" (UID: \"a482b42d-8782-433f-a05f-391d4516a850\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" Nov 02 18:17:03.110104 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.110070 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a482b42d-8782-433f-a05f-391d4516a850-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-5hhwn\" (UID: \"a482b42d-8782-433f-a05f-391d4516a850\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" Nov 02 18:17:03.110104 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.110095 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c40d5295-d354-4be9-89b8-6bd96ae2e056-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-f6kf2\" (UID: \"c40d5295-d354-4be9-89b8-6bd96ae2e056\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" Nov 02 18:17:03.110237 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.110135 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcx7f\" (UniqueName: \"kubernetes.io/projected/a482b42d-8782-433f-a05f-391d4516a850-kube-api-access-dcx7f\") pod \"maas-default-gateway-openshift-default-76f4876667-5hhwn\" (UID: \"a482b42d-8782-433f-a05f-391d4516a850\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" Nov 02 18:17:03.110237 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.110155 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6mvw\" (UniqueName: \"kubernetes.io/projected/c40d5295-d354-4be9-89b8-6bd96ae2e056-kube-api-access-v6mvw\") pod \"maas-default-gateway-openshift-default-76f4876667-f6kf2\" (UID: \"c40d5295-d354-4be9-89b8-6bd96ae2e056\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" Nov 02 18:17:03.190740 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.190700 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" Nov 02 18:17:03.222740 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.222715 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" Nov 02 18:17:03.500920 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:17:03.500888 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda482b42d_8782_433f_a05f_391d4516a850.slice/crio-341f574dcf7acc255990f797195b4410381563bdcb6930fd6afdd5a22313e0ba WatchSource:0}: Error finding container 341f574dcf7acc255990f797195b4410381563bdcb6930fd6afdd5a22313e0ba: Status 404 returned error can't find the container with id 341f574dcf7acc255990f797195b4410381563bdcb6930fd6afdd5a22313e0ba Nov 02 18:17:03.501149 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.501125 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn"] Nov 02 18:17:03.503609 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.503579 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:17:03.503694 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.503642 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:17:03.503694 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.503669 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:17:03.535489 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:17:03.535461 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc40d5295_d354_4be9_89b8_6bd96ae2e056.slice/crio-1473e3e63df43a1f1404d736766e4dfb912cc2f6c5a7164d95f69358316df777 WatchSource:0}: Error finding container 1473e3e63df43a1f1404d736766e4dfb912cc2f6c5a7164d95f69358316df777: Status 404 returned error can't find the container with id 1473e3e63df43a1f1404d736766e4dfb912cc2f6c5a7164d95f69358316df777 Nov 02 18:17:03.535885 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.535863 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2"] Nov 02 18:17:03.537898 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.537867 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:17:03.537979 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.537931 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:17:03.537979 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.537962 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:17:03.792482 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.792441 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" event={"ID":"a482b42d-8782-433f-a05f-391d4516a850","Type":"ContainerStarted","Data":"2ae760ce770a74094a672c945b7ea02aa6840a286ef89d4956e465174bea5f55"} Nov 02 18:17:03.793010 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.792661 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" event={"ID":"a482b42d-8782-433f-a05f-391d4516a850","Type":"ContainerStarted","Data":"341f574dcf7acc255990f797195b4410381563bdcb6930fd6afdd5a22313e0ba"} Nov 02 18:17:03.795093 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.795059 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" event={"ID":"c40d5295-d354-4be9-89b8-6bd96ae2e056","Type":"ContainerStarted","Data":"121373a0bd4036ab1217ce39c4e5e3f025ee566118c3375f839fcc160f9f4ba1"} Nov 02 18:17:03.795250 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.795111 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" event={"ID":"c40d5295-d354-4be9-89b8-6bd96ae2e056","Type":"ContainerStarted","Data":"1473e3e63df43a1f1404d736766e4dfb912cc2f6c5a7164d95f69358316df777"} Nov 02 18:17:03.839802 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.839748 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" podStartSLOduration=1.839732944 podStartE2EDuration="1.839732944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:17:03.837773546 +0000 UTC m=+2841.533129058" watchObservedRunningTime="2026-11-02 18:17:03.839732944 +0000 UTC m=+2841.535088416" Nov 02 18:17:03.885165 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:03.885102 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" podStartSLOduration=1.885082376 podStartE2EDuration="1.885082376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:17:03.883590081 +0000 UTC m=+2841.578945554" watchObservedRunningTime="2026-11-02 18:17:03.885082376 +0000 UTC m=+2841.580437849" Nov 02 18:17:04.193866 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:04.193355 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" Nov 02 18:17:04.193866 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:04.193493 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5hhwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Nov 02 18:17:04.193866 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:04.193534 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" podUID="a482b42d-8782-433f-a05f-391d4516a850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Nov 02 18:17:04.222992 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:04.222960 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" Nov 02 18:17:04.224287 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:04.224263 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-f6kf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Nov 02 18:17:04.224417 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:04.224328 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" podUID="c40d5295-d354-4be9-89b8-6bd96ae2e056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Nov 02 18:17:05.193330 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:05.192737 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5hhwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Nov 02 18:17:05.193330 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:05.192808 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" podUID="a482b42d-8782-433f-a05f-391d4516a850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Nov 02 18:17:05.223012 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:05.222979 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-f6kf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Nov 02 18:17:05.223194 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:05.223030 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" podUID="c40d5295-d354-4be9-89b8-6bd96ae2e056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Nov 02 18:17:06.195330 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:06.194750 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5hhwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Nov 02 18:17:06.195330 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:06.194817 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" podUID="a482b42d-8782-433f-a05f-391d4516a850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Nov 02 18:17:06.223837 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:06.223803 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-f6kf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Nov 02 18:17:06.224087 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:06.224062 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" podUID="c40d5295-d354-4be9-89b8-6bd96ae2e056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Nov 02 18:17:07.191767 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:07.191722 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5hhwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Nov 02 18:17:07.192079 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:07.192050 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" podUID="a482b42d-8782-433f-a05f-391d4516a850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Nov 02 18:17:07.223868 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:07.223835 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-f6kf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Nov 02 18:17:07.224526 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:07.224497 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" podUID="c40d5295-d354-4be9-89b8-6bd96ae2e056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Nov 02 18:17:08.192186 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:08.191716 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5hhwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Nov 02 18:17:08.192186 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:08.191796 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" podUID="a482b42d-8782-433f-a05f-391d4516a850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Nov 02 18:17:08.223625 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:08.223589 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-f6kf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Nov 02 18:17:08.223792 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:08.223652 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" podUID="c40d5295-d354-4be9-89b8-6bd96ae2e056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Nov 02 18:17:09.197330 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:09.196720 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5hhwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Nov 02 18:17:09.197330 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:09.196789 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" podUID="a482b42d-8782-433f-a05f-391d4516a850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Nov 02 18:17:09.224066 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:09.224030 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-f6kf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Nov 02 18:17:09.224269 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:09.224088 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" podUID="c40d5295-d354-4be9-89b8-6bd96ae2e056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Nov 02 18:17:09.226812 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:17:09.226789 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:17:10.191992 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:10.191951 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5hhwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Nov 02 18:17:10.192179 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:10.192022 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" podUID="a482b42d-8782-433f-a05f-391d4516a850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Nov 02 18:17:10.223035 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:10.223002 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-f6kf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Nov 02 18:17:10.223425 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:10.223074 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" podUID="c40d5295-d354-4be9-89b8-6bd96ae2e056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Nov 02 18:17:11.782446 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:11.782406 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" Nov 02 18:17:11.783692 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:11.783660 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" Nov 02 18:17:11.784422 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:11.784399 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" Nov 02 18:17:11.842443 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:11.842410 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" Nov 02 18:17:11.843231 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:11.843200 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" Nov 02 18:17:11.844231 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:11.844198 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" Nov 02 18:17:13.797537 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:13.797502 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-67b9f44df6-zft8x" Nov 02 18:17:39.181748 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:17:39.181705 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:17:40.766845 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:40.766805 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:17:57.843020 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:17:57.842984 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:18:02.578126 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:18:02.578087 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:18:09.183858 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:18:09.183817 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:18:20.976507 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:18:20.976468 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:18:36.872797 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:18:36.872759 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:18:39.295080 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:18:39.295028 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:18:59.354211 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:18:59.354176 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:19:09.182948 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:19:09.182907 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:19:09.676662 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:19:09.676622 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:19:14.338967 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:19:14.338931 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:19:31.358861 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:19:31.358825 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:19:36.630549 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:19:36.630509 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:19:39.182151 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:19:39.182108 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:19:52.881651 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:19:52.881615 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:19:57.554581 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:19:57.554498 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:20:00.387465 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:20:00.387419 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-k97sr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:20:00.388471 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:20:00.388080 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-k97sr" podUID="ef3e0f41-024e-4c23-80c4-5cc5b5d08331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:20:00.441331 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:20:00.440449 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-cv4r2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:20:00.441331 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:20:00.440528 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cv4r2" podUID="6c0a1b02-6f85-4e92-be26-a9f0cb390cef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:20:09.182881 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:20:09.182845 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:20:13.085136 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:20:13.085102 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:20:18.494341 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:20:18.493685 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:20:33.258741 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:20:33.258693 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:20:39.182440 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:20:39.182403 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:20:45.077149 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:20:45.077114 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:20:51.649916 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:20:51.649883 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:21:09.181784 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:21:09.181740 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:21:16.454939 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:21:16.454904 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:21:33.445273 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:21:33.445184 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:21:39.182744 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:21:39.182705 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:21:43.582802 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:21:43.582757 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:22:09.182832 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:22:09.182788 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:22:13.068980 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:13.068948 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ce839a-dbf4b7dd9lqwbj"] Nov 02 18:22:13.071717 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:13.071699 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce839a-dbf4b7dd9lqwbj" Nov 02 18:22:13.090519 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:13.090491 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-ce839a\"" Nov 02 18:22:13.093821 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:13.093791 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ce839a-dbf4b7dd9lqwbj"] Nov 02 18:22:13.132829 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:13.132796 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce839a-547457j2sm"] Nov 02 18:22:13.133269 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:13.133247 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0967d811-ee4e-45ad-8503-c030a2ccbef7-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ce839a-dbf4b7dd9lqwbj\" (UID: \"0967d811-ee4e-45ad-8503-c030a2ccbef7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce839a-dbf4b7dd9lqwbj" Nov 02 18:22:13.133346 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:13.133286 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrhjz\" (UniqueName: \"kubernetes.io/projected/0967d811-ee4e-45ad-8503-c030a2ccbef7-kube-api-access-hrhjz\") pod \"payload-processing-e2e-worker-single-main-ce839a-dbf4b7dd9lqwbj\" (UID: \"0967d811-ee4e-45ad-8503-c030a2ccbef7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce839a-dbf4b7dd9lqwbj" Nov 02 18:22:13.135651 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:13.135635 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce839a-547457j2sm" Nov 02 18:22:13.156741 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:13.156687 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce839a-547457j2sm"] Nov 02 18:22:13.156924 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:13.156886 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-07eb42ea-dockercfg-nfk7f\"" Nov 02 18:22:13.236014 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:13.235978 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c7hd\" (UniqueName: \"kubernetes.io/projected/54fc7150-2d04-4080-be64-7a15b6344f1c-kube-api-access-2c7hd\") pod \"payload-pre-processing-e2e-worker-single-main-ce839a-547457j2sm\" (UID: \"54fc7150-2d04-4080-be64-7a15b6344f1c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce839a-547457j2sm" Nov 02 18:22:13.236222 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:13.236184 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0967d811-ee4e-45ad-8503-c030a2ccbef7-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ce839a-dbf4b7dd9lqwbj\" (UID: \"0967d811-ee4e-45ad-8503-c030a2ccbef7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce839a-dbf4b7dd9lqwbj" Nov 02 18:22:13.236360 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:13.236247 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrhjz\" (UniqueName: \"kubernetes.io/projected/0967d811-ee4e-45ad-8503-c030a2ccbef7-kube-api-access-hrhjz\") pod \"payload-processing-e2e-worker-single-main-ce839a-dbf4b7dd9lqwbj\" (UID: \"0967d811-ee4e-45ad-8503-c030a2ccbef7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce839a-dbf4b7dd9lqwbj" Nov 02 18:22:13.236360 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:13.236336 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54fc7150-2d04-4080-be64-7a15b6344f1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ce839a-547457j2sm\" (UID: \"54fc7150-2d04-4080-be64-7a15b6344f1c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce839a-547457j2sm" Nov 02 18:22:13.237233 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:13.237206 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0967d811-ee4e-45ad-8503-c030a2ccbef7-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ce839a-dbf4b7dd9lqwbj\" (UID: \"0967d811-ee4e-45ad-8503-c030a2ccbef7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce839a-dbf4b7dd9lqwbj" Nov 02 18:22:13.261741 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:13.261713 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrhjz\" (UniqueName: \"kubernetes.io/projected/0967d811-ee4e-45ad-8503-c030a2ccbef7-kube-api-access-hrhjz\") pod \"payload-processing-e2e-worker-single-main-ce839a-dbf4b7dd9lqwbj\" (UID: \"0967d811-ee4e-45ad-8503-c030a2ccbef7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce839a-dbf4b7dd9lqwbj" Nov 02 18:22:13.337076 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:13.336998 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54fc7150-2d04-4080-be64-7a15b6344f1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ce839a-547457j2sm\" (UID: \"54fc7150-2d04-4080-be64-7a15b6344f1c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce839a-547457j2sm" Nov 02 18:22:13.337076 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:13.337064 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c7hd\" (UniqueName: \"kubernetes.io/projected/54fc7150-2d04-4080-be64-7a15b6344f1c-kube-api-access-2c7hd\") pod \"payload-pre-processing-e2e-worker-single-main-ce839a-547457j2sm\" (UID: \"54fc7150-2d04-4080-be64-7a15b6344f1c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce839a-547457j2sm" Nov 02 18:22:13.337805 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:13.337777 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54fc7150-2d04-4080-be64-7a15b6344f1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ce839a-547457j2sm\" (UID: \"54fc7150-2d04-4080-be64-7a15b6344f1c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce839a-547457j2sm" Nov 02 18:22:13.364055 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:13.364027 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c7hd\" (UniqueName: \"kubernetes.io/projected/54fc7150-2d04-4080-be64-7a15b6344f1c-kube-api-access-2c7hd\") pod \"payload-pre-processing-e2e-worker-single-main-ce839a-547457j2sm\" (UID: \"54fc7150-2d04-4080-be64-7a15b6344f1c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce839a-547457j2sm" Nov 02 18:22:13.383887 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:13.383862 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce839a-dbf4b7dd9lqwbj" Nov 02 18:22:13.449349 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:13.448678 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce839a-547457j2sm" Nov 02 18:22:13.677716 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:22:13.677687 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0967d811_ee4e_45ad_8503_c030a2ccbef7.slice/crio-7a949ce45fbc5cb0f797ee9ded05ad5583d289cfc716c184a012e15be3ffd6ee WatchSource:0}: Error finding container 7a949ce45fbc5cb0f797ee9ded05ad5583d289cfc716c184a012e15be3ffd6ee: Status 404 returned error can't find the container with id 7a949ce45fbc5cb0f797ee9ded05ad5583d289cfc716c184a012e15be3ffd6ee Nov 02 18:22:13.678666 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:13.678644 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ce839a-dbf4b7dd9lqwbj"] Nov 02 18:22:13.762050 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:13.761650 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce839a-547457j2sm"] Nov 02 18:22:13.775149 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:22:13.775112 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54fc7150_2d04_4080_be64_7a15b6344f1c.slice/crio-28418bcb8e34ce95c275646b6fc39622904d0370af48dcaf7082342ac60a3910 WatchSource:0}: Error finding container 28418bcb8e34ce95c275646b6fc39622904d0370af48dcaf7082342ac60a3910: Status 404 returned error can't find the container with id 28418bcb8e34ce95c275646b6fc39622904d0370af48dcaf7082342ac60a3910 Nov 02 18:22:14.590393 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:14.590353 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce839a-dbf4b7dd9lqwbj" event={"ID":"0967d811-ee4e-45ad-8503-c030a2ccbef7","Type":"ContainerStarted","Data":"7c66667ad281d9d1cdeb0194d25525f8107be93e3f819626b3c5d6fd5e76dda4"} Nov 02 18:22:14.591003 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:14.590410 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce839a-dbf4b7dd9lqwbj" event={"ID":"0967d811-ee4e-45ad-8503-c030a2ccbef7","Type":"ContainerStarted","Data":"7a949ce45fbc5cb0f797ee9ded05ad5583d289cfc716c184a012e15be3ffd6ee"} Nov 02 18:22:14.591003 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:14.590455 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce839a-dbf4b7dd9lqwbj" Nov 02 18:22:14.592077 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:14.592053 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce839a-547457j2sm" event={"ID":"54fc7150-2d04-4080-be64-7a15b6344f1c","Type":"ContainerStarted","Data":"efae3fcb316d7fa484d059a051ae4fd4854f8e8683510fc40e57c14330aadd7a"} Nov 02 18:22:14.592195 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:14.592089 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce839a-547457j2sm" event={"ID":"54fc7150-2d04-4080-be64-7a15b6344f1c","Type":"ContainerStarted","Data":"28418bcb8e34ce95c275646b6fc39622904d0370af48dcaf7082342ac60a3910"} Nov 02 18:22:14.592195 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:14.592182 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce839a-547457j2sm" Nov 02 18:22:14.635717 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:14.635664 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce839a-dbf4b7dd9lqwbj" podStartSLOduration=1.635651172 podStartE2EDuration="1.635651172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:22:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:22:14.634165252 +0000 UTC m=+3152.329520755" watchObservedRunningTime="2026-11-02 18:22:14.635651172 +0000 UTC m=+3152.331006672" Nov 02 18:22:14.676658 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:14.676614 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce839a-547457j2sm" podStartSLOduration=1.6766022459999999 podStartE2EDuration="1.676602246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:22:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:22:14.67471264 +0000 UTC m=+3152.370068131" watchObservedRunningTime="2026-11-02 18:22:14.676602246 +0000 UTC m=+3152.371957718" Nov 02 18:22:16.983618 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:16.983579 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:22:25.599292 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:25.599257 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce839a-dbf4b7dd9lqwbj" Nov 02 18:22:25.599736 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:25.599348 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce839a-547457j2sm" Nov 02 18:22:30.881784 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:30.881747 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:22:39.183546 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:22:39.183503 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:22:46.697274 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:22:46.697236 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:23:09.181808 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:23:09.181756 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:23:13.146975 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:13.146937 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:23:22.324475 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:22.324437 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ce839a-dbf4b7dd9lqwbj"] Nov 02 18:23:22.356530 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:22.356492 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce839a-547457j2sm"] Nov 02 18:23:25.377566 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:25.377531 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:23:27.324782 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:27.324739 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce839a-dbf4b7dd9lqwbj" podUID="0967d811-ee4e-45ad-8503-c030a2ccbef7" containerName="payload-processing" containerID="cri-o://7c66667ad281d9d1cdeb0194d25525f8107be93e3f819626b3c5d6fd5e76dda4" gracePeriod=25 Nov 02 18:23:27.357423 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:27.357370 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce839a-547457j2sm" podUID="54fc7150-2d04-4080-be64-7a15b6344f1c" containerName="payload-pre-processing" containerID="cri-o://efae3fcb316d7fa484d059a051ae4fd4854f8e8683510fc40e57c14330aadd7a" gracePeriod=25 Nov 02 18:23:35.598289 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:35.598206 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce839a-547457j2sm" podUID="54fc7150-2d04-4080-be64-7a15b6344f1c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:23:35.598684 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:35.598218 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce839a-dbf4b7dd9lqwbj" podUID="0967d811-ee4e-45ad-8503-c030a2ccbef7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:23:39.183054 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:23:39.183016 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:23:45.598070 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:45.598033 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce839a-dbf4b7dd9lqwbj" podUID="0967d811-ee4e-45ad-8503-c030a2ccbef7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:23:45.598070 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:45.598060 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce839a-547457j2sm" podUID="54fc7150-2d04-4080-be64-7a15b6344f1c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:23:47.614373 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:47.612408 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce839a-dbf4b7dd9lqwbj" Nov 02 18:23:47.636083 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:47.636060 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce839a-547457j2sm" Nov 02 18:23:47.762073 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:47.762042 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0967d811-ee4e-45ad-8503-c030a2ccbef7-kube-api-access-hrhjz\" (UniqueName: \"kubernetes.io/projected/0967d811-ee4e-45ad-8503-c030a2ccbef7-kube-api-access-hrhjz\") pod \"0967d811-ee4e-45ad-8503-c030a2ccbef7\" (UID: \"0967d811-ee4e-45ad-8503-c030a2ccbef7\") " Nov 02 18:23:47.762241 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:47.762153 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0967d811-ee4e-45ad-8503-c030a2ccbef7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0967d811-ee4e-45ad-8503-c030a2ccbef7-plugins-config-volume\") pod \"0967d811-ee4e-45ad-8503-c030a2ccbef7\" (UID: \"0967d811-ee4e-45ad-8503-c030a2ccbef7\") " Nov 02 18:23:47.762241 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:47.762195 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54fc7150-2d04-4080-be64-7a15b6344f1c-kube-api-access-2c7hd\" (UniqueName: \"kubernetes.io/projected/54fc7150-2d04-4080-be64-7a15b6344f1c-kube-api-access-2c7hd\") pod \"54fc7150-2d04-4080-be64-7a15b6344f1c\" (UID: \"54fc7150-2d04-4080-be64-7a15b6344f1c\") " Nov 02 18:23:47.762357 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:47.762253 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54fc7150-2d04-4080-be64-7a15b6344f1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54fc7150-2d04-4080-be64-7a15b6344f1c-plugins-config-volume\") pod \"54fc7150-2d04-4080-be64-7a15b6344f1c\" (UID: \"54fc7150-2d04-4080-be64-7a15b6344f1c\") " Nov 02 18:23:47.762715 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:47.762674 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0967d811-ee4e-45ad-8503-c030a2ccbef7-plugins-config-volume" pod "0967d811-ee4e-45ad-8503-c030a2ccbef7" (UID: "0967d811-ee4e-45ad-8503-c030a2ccbef7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:23:47.762865 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:47.762728 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54fc7150-2d04-4080-be64-7a15b6344f1c-plugins-config-volume" pod "54fc7150-2d04-4080-be64-7a15b6344f1c" (UID: "54fc7150-2d04-4080-be64-7a15b6344f1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:23:47.764384 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:47.764357 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0967d811-ee4e-45ad-8503-c030a2ccbef7-kube-api-access-hrhjz" pod "0967d811-ee4e-45ad-8503-c030a2ccbef7" (UID: "0967d811-ee4e-45ad-8503-c030a2ccbef7"). InnerVolumeSpecName "kube-api-access-hrhjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:23:47.764496 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:47.764396 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54fc7150-2d04-4080-be64-7a15b6344f1c-kube-api-access-2c7hd" pod "54fc7150-2d04-4080-be64-7a15b6344f1c" (UID: "54fc7150-2d04-4080-be64-7a15b6344f1c"). InnerVolumeSpecName "kube-api-access-2c7hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:23:47.863877 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:47.863802 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54fc7150-2d04-4080-be64-7a15b6344f1c-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:23:47.863877 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:47.863831 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrhjz\" (UniqueName: \"kubernetes.io/projected/0967d811-ee4e-45ad-8503-c030a2ccbef7-kube-api-access-hrhjz\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:23:47.863877 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:47.863840 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0967d811-ee4e-45ad-8503-c030a2ccbef7-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:23:47.863877 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:47.863849 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c7hd\" (UniqueName: \"kubernetes.io/projected/54fc7150-2d04-4080-be64-7a15b6344f1c-kube-api-access-2c7hd\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:23:48.043874 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:48.043841 4471 generic.go:356] "Generic (PLEG): container finished" podID="54fc7150-2d04-4080-be64-7a15b6344f1c" containerID="efae3fcb316d7fa484d059a051ae4fd4854f8e8683510fc40e57c14330aadd7a" exitCode=0 Nov 02 18:23:48.044072 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:48.043889 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce839a-547457j2sm" Nov 02 18:23:48.044072 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:48.043923 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce839a-547457j2sm" event={"ID":"54fc7150-2d04-4080-be64-7a15b6344f1c","Type":"ContainerDied","Data":"efae3fcb316d7fa484d059a051ae4fd4854f8e8683510fc40e57c14330aadd7a"} Nov 02 18:23:48.044072 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:48.044009 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce839a-547457j2sm" event={"ID":"54fc7150-2d04-4080-be64-7a15b6344f1c","Type":"ContainerDied","Data":"28418bcb8e34ce95c275646b6fc39622904d0370af48dcaf7082342ac60a3910"} Nov 02 18:23:48.044072 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:48.044025 4471 scope.go:122] "RemoveContainer" containerID="efae3fcb316d7fa484d059a051ae4fd4854f8e8683510fc40e57c14330aadd7a" Nov 02 18:23:48.045342 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:48.045316 4471 generic.go:356] "Generic (PLEG): container finished" podID="0967d811-ee4e-45ad-8503-c030a2ccbef7" containerID="7c66667ad281d9d1cdeb0194d25525f8107be93e3f819626b3c5d6fd5e76dda4" exitCode=0 Nov 02 18:23:48.045421 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:48.045333 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce839a-dbf4b7dd9lqwbj" event={"ID":"0967d811-ee4e-45ad-8503-c030a2ccbef7","Type":"ContainerDied","Data":"7c66667ad281d9d1cdeb0194d25525f8107be93e3f819626b3c5d6fd5e76dda4"} Nov 02 18:23:48.045421 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:48.045358 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce839a-dbf4b7dd9lqwbj" Nov 02 18:23:48.045421 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:48.045388 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce839a-dbf4b7dd9lqwbj" event={"ID":"0967d811-ee4e-45ad-8503-c030a2ccbef7","Type":"ContainerDied","Data":"7a949ce45fbc5cb0f797ee9ded05ad5583d289cfc716c184a012e15be3ffd6ee"} Nov 02 18:23:48.053016 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:48.052999 4471 scope.go:122] "RemoveContainer" containerID="efae3fcb316d7fa484d059a051ae4fd4854f8e8683510fc40e57c14330aadd7a" Nov 02 18:23:48.053316 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:23:48.053280 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efae3fcb316d7fa484d059a051ae4fd4854f8e8683510fc40e57c14330aadd7a\": container with ID starting with efae3fcb316d7fa484d059a051ae4fd4854f8e8683510fc40e57c14330aadd7a not found: ID does not exist" containerID="efae3fcb316d7fa484d059a051ae4fd4854f8e8683510fc40e57c14330aadd7a" Nov 02 18:23:48.053400 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:48.053328 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efae3fcb316d7fa484d059a051ae4fd4854f8e8683510fc40e57c14330aadd7a"} err="failed to get container status \"efae3fcb316d7fa484d059a051ae4fd4854f8e8683510fc40e57c14330aadd7a\": rpc error: code = NotFound desc = could not find container \"efae3fcb316d7fa484d059a051ae4fd4854f8e8683510fc40e57c14330aadd7a\": container with ID starting with efae3fcb316d7fa484d059a051ae4fd4854f8e8683510fc40e57c14330aadd7a not found: ID does not exist" Nov 02 18:23:48.053400 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:48.053351 4471 scope.go:122] "RemoveContainer" containerID="7c66667ad281d9d1cdeb0194d25525f8107be93e3f819626b3c5d6fd5e76dda4" Nov 02 18:23:48.061160 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:48.061142 4471 scope.go:122] "RemoveContainer" containerID="7c66667ad281d9d1cdeb0194d25525f8107be93e3f819626b3c5d6fd5e76dda4" Nov 02 18:23:48.061427 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:23:48.061409 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c66667ad281d9d1cdeb0194d25525f8107be93e3f819626b3c5d6fd5e76dda4\": container with ID starting with 7c66667ad281d9d1cdeb0194d25525f8107be93e3f819626b3c5d6fd5e76dda4 not found: ID does not exist" containerID="7c66667ad281d9d1cdeb0194d25525f8107be93e3f819626b3c5d6fd5e76dda4" Nov 02 18:23:48.061481 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:48.061434 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c66667ad281d9d1cdeb0194d25525f8107be93e3f819626b3c5d6fd5e76dda4"} err="failed to get container status \"7c66667ad281d9d1cdeb0194d25525f8107be93e3f819626b3c5d6fd5e76dda4\": rpc error: code = NotFound desc = could not find container \"7c66667ad281d9d1cdeb0194d25525f8107be93e3f819626b3c5d6fd5e76dda4\": container with ID starting with 7c66667ad281d9d1cdeb0194d25525f8107be93e3f819626b3c5d6fd5e76dda4 not found: ID does not exist" Nov 02 18:23:48.096885 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:48.096861 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce839a-547457j2sm"] Nov 02 18:23:48.109461 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:48.109432 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce839a-547457j2sm"] Nov 02 18:23:48.136723 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:48.136662 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ce839a-dbf4b7dd9lqwbj"] Nov 02 18:23:48.143461 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:48.143441 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ce839a-dbf4b7dd9lqwbj"] Nov 02 18:23:48.438911 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:48.438837 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0967d811-ee4e-45ad-8503-c030a2ccbef7" path="/var/lib/kubelet/pods/0967d811-ee4e-45ad-8503-c030a2ccbef7/volumes" Nov 02 18:23:48.439295 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:23:48.439274 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54fc7150-2d04-4080-be64-7a15b6344f1c" path="/var/lib/kubelet/pods/54fc7150-2d04-4080-be64-7a15b6344f1c/volumes" Nov 02 18:24:09.182679 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:24:09.182629 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:24:39.182357 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:24:39.182318 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:25:09.182548 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:25:09.182504 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:25:14.665788 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:25:14.665742 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:25:30.257184 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:25:30.257147 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:25:35.482331 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:25:35.481482 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:25:39.182753 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:25:39.182714 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:25:51.566084 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:25:51.566048 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:25:58.346205 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:25:58.346112 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe6df-7c7d8m8zwg"] Nov 02 18:25:58.346870 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:25:58.346845 4471 cpu_manager.go:414] "containerMap: removing container" podUID="54fc7150-2d04-4080-be64-7a15b6344f1c" containerName="payload-pre-processing" Nov 02 18:25:58.346870 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:25:58.346871 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54fc7150-2d04-4080-be64-7a15b6344f1c" containerName="payload-pre-processing" Nov 02 18:25:58.347032 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:25:58.346895 4471 cpu_manager.go:414] "containerMap: removing container" podUID="0967d811-ee4e-45ad-8503-c030a2ccbef7" containerName="payload-processing" Nov 02 18:25:58.347032 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:25:58.346901 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0967d811-ee4e-45ad-8503-c030a2ccbef7" containerName="payload-processing" Nov 02 18:25:58.347032 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:25:58.346977 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="54fc7150-2d04-4080-be64-7a15b6344f1c" containerName="payload-pre-processing" Nov 02 18:25:58.347032 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:25:58.346988 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="0967d811-ee4e-45ad-8503-c030a2ccbef7" containerName="payload-processing" Nov 02 18:25:58.350099 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:25:58.350082 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe6df-7c7d8m8zwg" Nov 02 18:25:58.369863 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:25:58.369833 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe6df-7c7d8m8zwg"] Nov 02 18:25:58.379639 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:25:58.379614 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-dbe6df\"" Nov 02 18:25:58.379773 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:25:58.379662 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-3b72f8da-dockercfg-vltdq\"" Nov 02 18:25:58.433980 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:25:58.433935 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d27a806-894f-45a2-878c-9e958bba3982-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-dbe6df-7c7d8m8zwg\" (UID: \"6d27a806-894f-45a2-878c-9e958bba3982\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe6df-7c7d8m8zwg" Nov 02 18:25:58.435227 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:25:58.435202 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjlnr\" (UniqueName: \"kubernetes.io/projected/6d27a806-894f-45a2-878c-9e958bba3982-kube-api-access-hjlnr\") pod \"payload-pre-processing-e2e-worker-single-main-dbe6df-7c7d8m8zwg\" (UID: \"6d27a806-894f-45a2-878c-9e958bba3982\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe6df-7c7d8m8zwg" Nov 02 18:25:58.536376 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:25:58.536338 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjlnr\" (UniqueName: \"kubernetes.io/projected/6d27a806-894f-45a2-878c-9e958bba3982-kube-api-access-hjlnr\") pod \"payload-pre-processing-e2e-worker-single-main-dbe6df-7c7d8m8zwg\" (UID: \"6d27a806-894f-45a2-878c-9e958bba3982\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe6df-7c7d8m8zwg" Nov 02 18:25:58.536575 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:25:58.536412 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d27a806-894f-45a2-878c-9e958bba3982-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-dbe6df-7c7d8m8zwg\" (UID: \"6d27a806-894f-45a2-878c-9e958bba3982\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe6df-7c7d8m8zwg" Nov 02 18:25:58.537095 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:25:58.537076 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d27a806-894f-45a2-878c-9e958bba3982-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-dbe6df-7c7d8m8zwg\" (UID: \"6d27a806-894f-45a2-878c-9e958bba3982\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe6df-7c7d8m8zwg" Nov 02 18:25:58.560099 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:25:58.560070 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjlnr\" (UniqueName: \"kubernetes.io/projected/6d27a806-894f-45a2-878c-9e958bba3982-kube-api-access-hjlnr\") pod \"payload-pre-processing-e2e-worker-single-main-dbe6df-7c7d8m8zwg\" (UID: \"6d27a806-894f-45a2-878c-9e958bba3982\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe6df-7c7d8m8zwg" Nov 02 18:25:58.662996 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:25:58.662922 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe6df-7c7d8m8zwg" Nov 02 18:25:58.957607 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:25:58.957567 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d27a806_894f_45a2_878c_9e958bba3982.slice/crio-0c8433e721abf65aef926521e32a24275d290c3a2d5e36b0d4383213d304af40 WatchSource:0}: Error finding container 0c8433e721abf65aef926521e32a24275d290c3a2d5e36b0d4383213d304af40: Status 404 returned error can't find the container with id 0c8433e721abf65aef926521e32a24275d290c3a2d5e36b0d4383213d304af40 Nov 02 18:25:58.958101 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:25:58.957996 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe6df-7c7d8m8zwg"] Nov 02 18:25:59.787819 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:25:59.787778 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe6df-7c7d8m8zwg" event={"ID":"6d27a806-894f-45a2-878c-9e958bba3982","Type":"ContainerStarted","Data":"844f041ad86df8afb165714234d3995215e30ac88aa584935704acdd1ce4b43e"} Nov 02 18:25:59.788330 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:25:59.787833 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe6df-7c7d8m8zwg" event={"ID":"6d27a806-894f-45a2-878c-9e958bba3982","Type":"ContainerStarted","Data":"0c8433e721abf65aef926521e32a24275d290c3a2d5e36b0d4383213d304af40"} Nov 02 18:25:59.788330 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:25:59.787887 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe6df-7c7d8m8zwg" Nov 02 18:25:59.830144 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:25:59.830094 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe6df-7c7d8m8zwg" podStartSLOduration=1.830080722 podStartE2EDuration="1.830080722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:25:59.828774911 +0000 UTC m=+3377.524130417" watchObservedRunningTime="2026-11-02 18:25:59.830080722 +0000 UTC m=+3377.525436195" Nov 02 18:26:01.778247 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:26:01.778211 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:26:09.194891 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:26:09.194851 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:26:10.796393 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:26:10.796366 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe6df-7c7d8m8zwg" Nov 02 18:26:16.187076 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:26:16.187042 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:26:31.772517 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:26:31.772478 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:26:39.183060 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:26:39.183020 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:26:55.649843 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:26:55.649805 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:27:05.660816 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:27:05.660781 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe6df-7c7d8m8zwg"] Nov 02 18:27:09.182802 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:27:09.182761 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:27:10.460171 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:27:10.460100 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:27:10.662058 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:27:10.662011 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe6df-7c7d8m8zwg" podUID="6d27a806-894f-45a2-878c-9e958bba3982" containerName="payload-pre-processing" containerID="cri-o://844f041ad86df8afb165714234d3995215e30ac88aa584935704acdd1ce4b43e" gracePeriod=25 Nov 02 18:27:10.794205 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:27:10.794170 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe6df-7c7d8m8zwg" podUID="6d27a806-894f-45a2-878c-9e958bba3982" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:27:20.794559 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:27:20.794522 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe6df-7c7d8m8zwg" podUID="6d27a806-894f-45a2-878c-9e958bba3982" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:27:22.079836 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:27:22.079799 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:27:30.794044 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:27:30.794015 4471 logging.go:55] [core] [Channel #9861 SubChannel #9862]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.216:50052", ServerName: "10.133.0.216:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.216:50052: connect: connection refused" Nov 02 18:27:30.915612 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:27:30.915589 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe6df-7c7d8m8zwg" Nov 02 18:27:30.940756 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:27:30.940689 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d27a806-894f-45a2-878c-9e958bba3982-kube-api-access-hjlnr\" (UniqueName: \"kubernetes.io/projected/6d27a806-894f-45a2-878c-9e958bba3982-kube-api-access-hjlnr\") pod \"6d27a806-894f-45a2-878c-9e958bba3982\" (UID: \"6d27a806-894f-45a2-878c-9e958bba3982\") " Nov 02 18:27:30.940900 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:27:30.940782 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d27a806-894f-45a2-878c-9e958bba3982-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d27a806-894f-45a2-878c-9e958bba3982-plugins-config-volume\") pod \"6d27a806-894f-45a2-878c-9e958bba3982\" (UID: \"6d27a806-894f-45a2-878c-9e958bba3982\") " Nov 02 18:27:30.941179 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:27:30.941148 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d27a806-894f-45a2-878c-9e958bba3982-plugins-config-volume" pod "6d27a806-894f-45a2-878c-9e958bba3982" (UID: "6d27a806-894f-45a2-878c-9e958bba3982"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:27:30.942935 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:27:30.942906 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d27a806-894f-45a2-878c-9e958bba3982-kube-api-access-hjlnr" pod "6d27a806-894f-45a2-878c-9e958bba3982" (UID: "6d27a806-894f-45a2-878c-9e958bba3982"). InnerVolumeSpecName "kube-api-access-hjlnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:27:31.042576 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:27:31.042542 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjlnr\" (UniqueName: \"kubernetes.io/projected/6d27a806-894f-45a2-878c-9e958bba3982-kube-api-access-hjlnr\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:27:31.042576 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:27:31.042571 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d27a806-894f-45a2-878c-9e958bba3982-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:27:31.256289 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:27:31.256197 4471 generic.go:356] "Generic (PLEG): container finished" podID="6d27a806-894f-45a2-878c-9e958bba3982" containerID="844f041ad86df8afb165714234d3995215e30ac88aa584935704acdd1ce4b43e" exitCode=0 Nov 02 18:27:31.256289 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:27:31.256250 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe6df-7c7d8m8zwg" Nov 02 18:27:31.256529 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:27:31.256287 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe6df-7c7d8m8zwg" event={"ID":"6d27a806-894f-45a2-878c-9e958bba3982","Type":"ContainerDied","Data":"844f041ad86df8afb165714234d3995215e30ac88aa584935704acdd1ce4b43e"} Nov 02 18:27:31.256529 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:27:31.256376 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe6df-7c7d8m8zwg" event={"ID":"6d27a806-894f-45a2-878c-9e958bba3982","Type":"ContainerDied","Data":"0c8433e721abf65aef926521e32a24275d290c3a2d5e36b0d4383213d304af40"} Nov 02 18:27:31.256529 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:27:31.256401 4471 scope.go:122] "RemoveContainer" containerID="844f041ad86df8afb165714234d3995215e30ac88aa584935704acdd1ce4b43e" Nov 02 18:27:31.265869 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:27:31.265846 4471 scope.go:122] "RemoveContainer" containerID="844f041ad86df8afb165714234d3995215e30ac88aa584935704acdd1ce4b43e" Nov 02 18:27:31.266183 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:27:31.266161 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"844f041ad86df8afb165714234d3995215e30ac88aa584935704acdd1ce4b43e\": container with ID starting with 844f041ad86df8afb165714234d3995215e30ac88aa584935704acdd1ce4b43e not found: ID does not exist" containerID="844f041ad86df8afb165714234d3995215e30ac88aa584935704acdd1ce4b43e" Nov 02 18:27:31.266284 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:27:31.266192 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"844f041ad86df8afb165714234d3995215e30ac88aa584935704acdd1ce4b43e"} err="failed to get container status \"844f041ad86df8afb165714234d3995215e30ac88aa584935704acdd1ce4b43e\": rpc error: code = NotFound desc = could not find container \"844f041ad86df8afb165714234d3995215e30ac88aa584935704acdd1ce4b43e\": container with ID starting with 844f041ad86df8afb165714234d3995215e30ac88aa584935704acdd1ce4b43e not found: ID does not exist" Nov 02 18:27:31.308095 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:27:31.308068 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe6df-7c7d8m8zwg"] Nov 02 18:27:31.320625 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:27:31.320596 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe6df-7c7d8m8zwg"] Nov 02 18:27:31.793867 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:27:31.793827 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe6df-7c7d8m8zwg" podUID="6d27a806-894f-45a2-878c-9e958bba3982" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.216:50052\" within 1s: context deadline exceeded" Nov 02 18:27:32.438487 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:27:32.438454 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d27a806-894f-45a2-878c-9e958bba3982" path="/var/lib/kubelet/pods/6d27a806-894f-45a2-878c-9e958bba3982/volumes" Nov 02 18:27:39.182652 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:27:39.182616 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:27:39.275685 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:27:39.275642 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:27:42.833503 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:27:42.833458 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5hhwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:27:42.834292 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:27:42.834261 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" podUID="a482b42d-8782-433f-a05f-391d4516a850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:27:42.893457 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:27:42.893420 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-f6kf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:27:42.893775 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:27:42.893723 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" podUID="c40d5295-d354-4be9-89b8-6bd96ae2e056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:27:54.976520 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:27:54.976483 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:27:57.838481 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:27:57.838438 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5hhwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:27:57.839160 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:27:57.839126 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" podUID="a482b42d-8782-433f-a05f-391d4516a850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:27:57.888604 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:27:57.888507 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-f6kf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:27:57.888997 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:27:57.888957 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" podUID="c40d5295-d354-4be9-89b8-6bd96ae2e056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:28:07.169709 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:07.169672 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:28:09.336710 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:28:09.336636 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:28:15.780339 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:15.780287 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:28:24.757501 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:24.757467 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:28:32.262783 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:32.262749 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:28:39.182590 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:28:39.182552 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:28:48.162945 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:48.162910 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:28:56.534534 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.534447 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59"] Nov 02 18:28:56.534932 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.534920 4471 cpu_manager.go:414] "containerMap: removing container" podUID="6d27a806-894f-45a2-878c-9e958bba3982" containerName="payload-pre-processing" Nov 02 18:28:56.534979 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.534935 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d27a806-894f-45a2-878c-9e958bba3982" containerName="payload-pre-processing" Nov 02 18:28:56.535028 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.535011 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d27a806-894f-45a2-878c-9e958bba3982" containerName="payload-pre-processing" Nov 02 18:28:56.538352 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.538335 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:28:56.554586 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.554561 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ae417f29-openshift-default-dockercfg-s8dwn\"" Nov 02 18:28:56.569452 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.569426 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59"] Nov 02 18:28:56.664329 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.660954 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istio-podinfo\") pod \"e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:28:56.664329 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.661011 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istio-token\") pod \"e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:28:56.664329 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.661077 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istio-data\") pod \"e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:28:56.664329 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.661138 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-workload-certs\") pod \"e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:28:56.664329 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.661199 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzxpx\" (UniqueName: \"kubernetes.io/projected/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-kube-api-access-kzxpx\") pod \"e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:28:56.664329 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.662346 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-workload-socket\") pod \"e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:28:56.664329 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.662395 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istiod-ca-cert\") pod \"e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:28:56.664329 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.662431 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-credential-socket\") pod \"e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:28:56.664329 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.662462 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istio-envoy\") pod \"e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:28:56.763735 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.763700 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istio-data\") pod \"e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:28:56.763970 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.763754 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-workload-certs\") pod \"e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:28:56.763970 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.763793 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzxpx\" (UniqueName: \"kubernetes.io/projected/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-kube-api-access-kzxpx\") pod \"e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:28:56.763970 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.763817 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-workload-socket\") pod \"e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:28:56.763970 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.763840 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istiod-ca-cert\") pod \"e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:28:56.763970 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.763875 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-credential-socket\") pod \"e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:28:56.763970 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.763908 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istio-envoy\") pod \"e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:28:56.763970 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.763938 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istio-podinfo\") pod \"e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:28:56.763970 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.763970 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istio-token\") pod \"e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:28:56.764500 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.764165 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istio-data\") pod \"e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:28:56.764500 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.764236 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-workload-certs\") pod \"e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:28:56.764500 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.764278 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-workload-socket\") pod \"e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:28:56.764500 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.764433 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-credential-socket\") pod \"e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:28:56.764811 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.764786 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istiod-ca-cert\") pod \"e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:28:56.766535 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.766510 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istio-envoy\") pod \"e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:28:56.766657 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.766642 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istio-podinfo\") pod \"e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:28:56.793835 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.793768 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istio-token\") pod \"e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:28:56.793979 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.793837 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzxpx\" (UniqueName: \"kubernetes.io/projected/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-kube-api-access-kzxpx\") pod \"e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:28:56.852411 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:56.852381 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:28:57.162236 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:28:57.162202 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad2ffdb9_fb43_4f2c_a3ea_c0c7492694dc.slice/crio-109e1ab57acabcdda24378572380f9dabc1ffd229b37c8fc72a6e170c2188610 WatchSource:0}: Error finding container 109e1ab57acabcdda24378572380f9dabc1ffd229b37c8fc72a6e170c2188610: Status 404 returned error can't find the container with id 109e1ab57acabcdda24378572380f9dabc1ffd229b37c8fc72a6e170c2188610 Nov 02 18:28:57.162636 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:57.162607 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59"] Nov 02 18:28:57.164552 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:57.164511 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:28:57.164639 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:57.164578 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:28:57.164639 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:57.164607 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:28:57.989725 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:57.989690 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" event={"ID":"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc","Type":"ContainerStarted","Data":"2ca205fe00879d12add2d8d6aa20de9427f8b23c6af893a5baf44a5ccad0299f"} Nov 02 18:28:57.990126 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:57.989751 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" event={"ID":"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc","Type":"ContainerStarted","Data":"109e1ab57acabcdda24378572380f9dabc1ffd229b37c8fc72a6e170c2188610"} Nov 02 18:28:58.049791 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:58.049736 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" podStartSLOduration=2.049722587 podStartE2EDuration="2.049722587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:28:58.047677228 +0000 UTC m=+3555.743032702" watchObservedRunningTime="2026-11-02 18:28:58.049722587 +0000 UTC m=+3555.745078060" Nov 02 18:28:58.852600 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:58.852570 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:28:58.857575 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:58.857541 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:28:58.993809 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:58.993773 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:28:58.994782 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:28:58.994765 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:29:04.591340 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:04.591283 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-667c5f9d54-tq47s"] Nov 02 18:29:04.591794 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:04.591560 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-667c5f9d54-tq47s" podUID="d0158b72-5fdc-42dc-929d-d1e3663359f2" containerName="maas-api" containerID="cri-o://735e891456133727f3f7ab93b450fc96b15918a47815ee8d4ae733e86186b367" gracePeriod=30 Nov 02 18:29:05.463014 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:05.462977 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:29:05.849466 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:05.849440 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-667c5f9d54-tq47s" Nov 02 18:29:05.867010 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:05.866985 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0158b72-5fdc-42dc-929d-d1e3663359f2-kube-api-access-4cccx\" (UniqueName: \"kubernetes.io/projected/d0158b72-5fdc-42dc-929d-d1e3663359f2-kube-api-access-4cccx\") pod \"d0158b72-5fdc-42dc-929d-d1e3663359f2\" (UID: \"d0158b72-5fdc-42dc-929d-d1e3663359f2\") " Nov 02 18:29:05.867143 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:05.867040 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0158b72-5fdc-42dc-929d-d1e3663359f2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0158b72-5fdc-42dc-929d-d1e3663359f2-openshift-service-ca\") pod \"d0158b72-5fdc-42dc-929d-d1e3663359f2\" (UID: \"d0158b72-5fdc-42dc-929d-d1e3663359f2\") " Nov 02 18:29:05.867143 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:05.867063 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0158b72-5fdc-42dc-929d-d1e3663359f2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0158b72-5fdc-42dc-929d-d1e3663359f2-metrics-certs\") pod \"d0158b72-5fdc-42dc-929d-d1e3663359f2\" (UID: \"d0158b72-5fdc-42dc-929d-d1e3663359f2\") " Nov 02 18:29:05.867279 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:05.867162 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0158b72-5fdc-42dc-929d-d1e3663359f2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0158b72-5fdc-42dc-929d-d1e3663359f2-maas-api-tls\") pod \"d0158b72-5fdc-42dc-929d-d1e3663359f2\" (UID: \"d0158b72-5fdc-42dc-929d-d1e3663359f2\") " Nov 02 18:29:05.867506 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:05.867472 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0158b72-5fdc-42dc-929d-d1e3663359f2-openshift-service-ca" pod "d0158b72-5fdc-42dc-929d-d1e3663359f2" (UID: "d0158b72-5fdc-42dc-929d-d1e3663359f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:29:05.869215 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:05.869186 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0158b72-5fdc-42dc-929d-d1e3663359f2-kube-api-access-4cccx" pod "d0158b72-5fdc-42dc-929d-d1e3663359f2" (UID: "d0158b72-5fdc-42dc-929d-d1e3663359f2"). InnerVolumeSpecName "kube-api-access-4cccx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:05.869301 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:05.869272 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0158b72-5fdc-42dc-929d-d1e3663359f2-maas-api-tls" pod "d0158b72-5fdc-42dc-929d-d1e3663359f2" (UID: "d0158b72-5fdc-42dc-929d-d1e3663359f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:29:05.869301 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:05.869290 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0158b72-5fdc-42dc-929d-d1e3663359f2-metrics-certs" pod "d0158b72-5fdc-42dc-929d-d1e3663359f2" (UID: "d0158b72-5fdc-42dc-929d-d1e3663359f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:29:05.968203 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:05.968172 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0158b72-5fdc-42dc-929d-d1e3663359f2-maas-api-tls\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:29:05.968203 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:05.968199 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cccx\" (UniqueName: \"kubernetes.io/projected/d0158b72-5fdc-42dc-929d-d1e3663359f2-kube-api-access-4cccx\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:29:05.968203 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:05.968210 4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0158b72-5fdc-42dc-929d-d1e3663359f2-openshift-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:29:05.968458 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:05.968218 4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0158b72-5fdc-42dc-929d-d1e3663359f2-metrics-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:29:06.030362 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:06.030323 4471 generic.go:356] "Generic (PLEG): container finished" podID="d0158b72-5fdc-42dc-929d-d1e3663359f2" containerID="735e891456133727f3f7ab93b450fc96b15918a47815ee8d4ae733e86186b367" exitCode=0 Nov 02 18:29:06.030521 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:06.030365 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-667c5f9d54-tq47s" event={"ID":"d0158b72-5fdc-42dc-929d-d1e3663359f2","Type":"ContainerDied","Data":"735e891456133727f3f7ab93b450fc96b15918a47815ee8d4ae733e86186b367"} Nov 02 18:29:06.030521 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:06.030392 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-667c5f9d54-tq47s" Nov 02 18:29:06.030521 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:06.030426 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-667c5f9d54-tq47s" event={"ID":"d0158b72-5fdc-42dc-929d-d1e3663359f2","Type":"ContainerDied","Data":"4cd46e3ff7d71da3a9ed63b2ee7ec8bfaca96d74e9595ef7cbeec983432c2ad9"} Nov 02 18:29:06.030521 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:06.030449 4471 scope.go:122] "RemoveContainer" containerID="735e891456133727f3f7ab93b450fc96b15918a47815ee8d4ae733e86186b367" Nov 02 18:29:06.040001 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:06.039987 4471 scope.go:122] "RemoveContainer" containerID="735e891456133727f3f7ab93b450fc96b15918a47815ee8d4ae733e86186b367" Nov 02 18:29:06.040245 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:29:06.040228 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"735e891456133727f3f7ab93b450fc96b15918a47815ee8d4ae733e86186b367\": container with ID starting with 735e891456133727f3f7ab93b450fc96b15918a47815ee8d4ae733e86186b367 not found: ID does not exist" containerID="735e891456133727f3f7ab93b450fc96b15918a47815ee8d4ae733e86186b367" Nov 02 18:29:06.040297 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:06.040252 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735e891456133727f3f7ab93b450fc96b15918a47815ee8d4ae733e86186b367"} err="failed to get container status \"735e891456133727f3f7ab93b450fc96b15918a47815ee8d4ae733e86186b367\": rpc error: code = NotFound desc = could not find container \"735e891456133727f3f7ab93b450fc96b15918a47815ee8d4ae733e86186b367\": container with ID starting with 735e891456133727f3f7ab93b450fc96b15918a47815ee8d4ae733e86186b367 not found: ID does not exist" Nov 02 18:29:06.082545 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:06.082509 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-667c5f9d54-tq47s"] Nov 02 18:29:06.087997 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:06.087972 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-667c5f9d54-tq47s"] Nov 02 18:29:06.438983 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:06.438906 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0158b72-5fdc-42dc-929d-d1e3663359f2" path="/var/lib/kubelet/pods/d0158b72-5fdc-42dc-929d-d1e3663359f2/volumes" Nov 02 18:29:09.205746 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:29:09.205707 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:29:15.028620 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:15.028299 4471 patch_prober.go:29] interesting pod/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:29:15.028620 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:15.028503 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" podUID="ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:29:15.072770 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:15.072740 4471 generic.go:356] "Generic (PLEG): container finished" podID="ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc" containerID="2ca205fe00879d12add2d8d6aa20de9427f8b23c6af893a5baf44a5ccad0299f" exitCode=0 Nov 02 18:29:15.072914 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:15.072819 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" event={"ID":"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc","Type":"ContainerDied","Data":"2ca205fe00879d12add2d8d6aa20de9427f8b23c6af893a5baf44a5ccad0299f"} Nov 02 18:29:15.073190 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:15.073174 4471 scope.go:122] "RemoveContainer" containerID="2ca205fe00879d12add2d8d6aa20de9427f8b23c6af893a5baf44a5ccad0299f" Nov 02 18:29:15.074347 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:15.074296 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:29:15.074434 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:15.074383 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:29:15.074434 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:15.074413 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 18:29:15.602782 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:15.602740 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c"] Nov 02 18:29:15.603587 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:15.603559 4471 cpu_manager.go:414] "containerMap: removing container" podUID="d0158b72-5fdc-42dc-929d-d1e3663359f2" containerName="maas-api" Nov 02 18:29:15.603587 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:15.603587 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0158b72-5fdc-42dc-929d-d1e3663359f2" containerName="maas-api" Nov 02 18:29:15.603800 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:15.603727 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0158b72-5fdc-42dc-929d-d1e3663359f2" containerName="maas-api" Nov 02 18:29:15.606453 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:15.606419 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c" Nov 02 18:29:15.622591 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:15.622565 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 18:29:15.622713 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:15.622565 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 18:29:15.623812 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:15.623788 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c"] Nov 02 18:29:15.676328 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:15.675873 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d478a85-8b4a-4039-80d9-320a0beb4258-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c\" (UID: \"9d478a85-8b4a-4039-80d9-320a0beb4258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c" Nov 02 18:29:15.676328 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:15.675967 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d478a85-8b4a-4039-80d9-320a0beb4258-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c\" (UID: \"9d478a85-8b4a-4039-80d9-320a0beb4258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c" Nov 02 18:29:15.676328 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:15.676006 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g77bd\" (UniqueName: \"kubernetes.io/projected/9d478a85-8b4a-4039-80d9-320a0beb4258-kube-api-access-g77bd\") pod \"maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c\" (UID: \"9d478a85-8b4a-4039-80d9-320a0beb4258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c" Nov 02 18:29:15.676918 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:15.676894 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:29:15.677114 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:15.677100 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 18:29:15.778328 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:15.777914 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d478a85-8b4a-4039-80d9-320a0beb4258-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c\" (UID: \"9d478a85-8b4a-4039-80d9-320a0beb4258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c" Nov 02 18:29:15.778328 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:15.777985 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g77bd\" (UniqueName: \"kubernetes.io/projected/9d478a85-8b4a-4039-80d9-320a0beb4258-kube-api-access-g77bd\") pod \"maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c\" (UID: \"9d478a85-8b4a-4039-80d9-320a0beb4258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c" Nov 02 18:29:15.778328 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:15.778143 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d478a85-8b4a-4039-80d9-320a0beb4258-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c\" (UID: \"9d478a85-8b4a-4039-80d9-320a0beb4258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c" Nov 02 18:29:15.779058 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:15.779026 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d478a85-8b4a-4039-80d9-320a0beb4258-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c\" (UID: \"9d478a85-8b4a-4039-80d9-320a0beb4258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c" Nov 02 18:29:15.781528 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:15.781502 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d478a85-8b4a-4039-80d9-320a0beb4258-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c\" (UID: \"9d478a85-8b4a-4039-80d9-320a0beb4258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c" Nov 02 18:29:15.808752 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:29:15.808726 4471 projected.go:196] Error preparing data for projected volume kube-api-access-g77bd for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c: failed to fetch token: serviceaccounts "maas-api-cleanup" not found Nov 02 18:29:15.808886 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:29:15.808810 4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d478a85-8b4a-4039-80d9-320a0beb4258-kube-api-access-g77bd podName:9d478a85-8b4a-4039-80d9-320a0beb4258 nodeName:}" failed. No retries permitted until 2026-11-02 18:29:16.30878945 +0000 UTC m=+3574.004144916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-g77bd" (UniqueName: "kubernetes.io/projected/9d478a85-8b4a-4039-80d9-320a0beb4258-kube-api-access-g77bd") pod "maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c" (UID: "9d478a85-8b4a-4039-80d9-320a0beb4258") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found Nov 02 18:29:16.085134 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:16.085094 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" event={"ID":"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc","Type":"ContainerStarted","Data":"2746adfdf1544dea5f969617b39d07069d2670d4e342c50298b3bc36e8bcd4bf"} Nov 02 18:29:16.386414 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:16.386283 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g77bd\" (UniqueName: \"kubernetes.io/projected/9d478a85-8b4a-4039-80d9-320a0beb4258-kube-api-access-g77bd\") pod \"maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c\" (UID: \"9d478a85-8b4a-4039-80d9-320a0beb4258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c" Nov 02 18:29:16.402237 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:29:16.402208 4471 projected.go:196] Error preparing data for projected volume kube-api-access-g77bd for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c: failed to fetch token: serviceaccounts "maas-api-cleanup" not found Nov 02 18:29:16.402377 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:29:16.402277 4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d478a85-8b4a-4039-80d9-320a0beb4258-kube-api-access-g77bd podName:9d478a85-8b4a-4039-80d9-320a0beb4258 nodeName:}" failed. No retries permitted until 2026-11-02 18:29:17.40226187 +0000 UTC m=+3575.097617321 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-g77bd" (UniqueName: "kubernetes.io/projected/9d478a85-8b4a-4039-80d9-320a0beb4258-kube-api-access-g77bd") pod "maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c" (UID: "9d478a85-8b4a-4039-80d9-320a0beb4258") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found Nov 02 18:29:16.853360 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:16.853318 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:29:16.853545 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:16.853507 4471 patch_prober.go:29] interesting pod/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 02 18:29:16.853596 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:16.853553 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" podUID="ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 02 18:29:17.413095 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:17.413062 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g77bd\" (UniqueName: \"kubernetes.io/projected/9d478a85-8b4a-4039-80d9-320a0beb4258-kube-api-access-g77bd\") pod \"maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c\" (UID: \"9d478a85-8b4a-4039-80d9-320a0beb4258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c" Nov 02 18:29:17.436779 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:17.436748 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g77bd\" (UniqueName: \"kubernetes.io/projected/9d478a85-8b4a-4039-80d9-320a0beb4258-kube-api-access-g77bd\") pod \"maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c\" (UID: \"9d478a85-8b4a-4039-80d9-320a0beb4258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c" Nov 02 18:29:17.721125 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:17.721037 4471 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c" secret="" err="secret \"maas-api-cleanup-dockercfg-7flf7\" not found" Nov 02 18:29:17.721125 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:17.721107 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c" Nov 02 18:29:17.852904 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:17.852869 4471 patch_prober.go:29] interesting pod/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 02 18:29:17.853061 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:17.852930 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" podUID="ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 02 18:29:18.038249 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:29:18.038220 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d478a85_8b4a_4039_80d9_320a0beb4258.slice/crio-bca3af4cc29da6ad162bbb45ce1fa7f4ef2f6d647311987abbeea3864221d083 WatchSource:0}: Error finding container bca3af4cc29da6ad162bbb45ce1fa7f4ef2f6d647311987abbeea3864221d083: Status 404 returned error can't find the container with id bca3af4cc29da6ad162bbb45ce1fa7f4ef2f6d647311987abbeea3864221d083 Nov 02 18:29:18.038411 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:18.038371 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c"] Nov 02 18:29:18.095237 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:18.095203 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c" event={"ID":"9d478a85-8b4a-4039-80d9-320a0beb4258","Type":"ContainerStarted","Data":"bca3af4cc29da6ad162bbb45ce1fa7f4ef2f6d647311987abbeea3864221d083"} Nov 02 18:29:18.853745 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:18.853709 4471 patch_prober.go:29] interesting pod/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 02 18:29:18.854136 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:18.853767 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" podUID="ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 02 18:29:19.101730 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:19.101697 4471 generic.go:356] "Generic (PLEG): container finished" podID="9d478a85-8b4a-4039-80d9-320a0beb4258" containerID="46bd18fdac72fb99ed374062ade0bcf336e8ff80be25c19eae250e2b93538bb2" exitCode=0 Nov 02 18:29:19.101881 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:19.101743 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c" event={"ID":"9d478a85-8b4a-4039-80d9-320a0beb4258","Type":"ContainerDied","Data":"46bd18fdac72fb99ed374062ade0bcf336e8ff80be25c19eae250e2b93538bb2"} Nov 02 18:29:19.853539 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:19.853502 4471 patch_prober.go:29] interesting pod/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 02 18:29:19.853796 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:19.853556 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" podUID="ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 02 18:29:20.290345 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:20.290320 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c" Nov 02 18:29:20.342773 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:20.342737 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d478a85-8b4a-4039-80d9-320a0beb4258-kube-api-access-g77bd\" (UniqueName: \"kubernetes.io/projected/9d478a85-8b4a-4039-80d9-320a0beb4258-kube-api-access-g77bd\") pod \"9d478a85-8b4a-4039-80d9-320a0beb4258\" (UID: \"9d478a85-8b4a-4039-80d9-320a0beb4258\") " Nov 02 18:29:20.342968 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:20.342845 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d478a85-8b4a-4039-80d9-320a0beb4258-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d478a85-8b4a-4039-80d9-320a0beb4258-maas-api-openshift-ca\") pod \"9d478a85-8b4a-4039-80d9-320a0beb4258\" (UID: \"9d478a85-8b4a-4039-80d9-320a0beb4258\") " Nov 02 18:29:20.342968 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:20.342874 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d478a85-8b4a-4039-80d9-320a0beb4258-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d478a85-8b4a-4039-80d9-320a0beb4258-maas-api-xks-ca\") pod \"9d478a85-8b4a-4039-80d9-320a0beb4258\" (UID: \"9d478a85-8b4a-4039-80d9-320a0beb4258\") " Nov 02 18:29:20.343321 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:20.343276 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d478a85-8b4a-4039-80d9-320a0beb4258-maas-api-openshift-ca" pod "9d478a85-8b4a-4039-80d9-320a0beb4258" (UID: "9d478a85-8b4a-4039-80d9-320a0beb4258"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:29:20.345037 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:20.345006 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d478a85-8b4a-4039-80d9-320a0beb4258-maas-api-xks-ca" pod "9d478a85-8b4a-4039-80d9-320a0beb4258" (UID: "9d478a85-8b4a-4039-80d9-320a0beb4258"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:29:20.345134 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:20.345054 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d478a85-8b4a-4039-80d9-320a0beb4258-kube-api-access-g77bd" pod "9d478a85-8b4a-4039-80d9-320a0beb4258" (UID: "9d478a85-8b4a-4039-80d9-320a0beb4258"). InnerVolumeSpecName "kube-api-access-g77bd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:20.443693 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:20.443615 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g77bd\" (UniqueName: \"kubernetes.io/projected/9d478a85-8b4a-4039-80d9-320a0beb4258-kube-api-access-g77bd\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:29:20.443693 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:20.443645 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d478a85-8b4a-4039-80d9-320a0beb4258-maas-api-openshift-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:29:20.443693 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:20.443656 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d478a85-8b4a-4039-80d9-320a0beb4258-maas-api-xks-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:29:20.852765 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:20.852725 4471 patch_prober.go:29] interesting pod/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 02 18:29:20.852943 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:20.852790 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" podUID="ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 02 18:29:21.112938 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:21.112850 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c" event={"ID":"9d478a85-8b4a-4039-80d9-320a0beb4258","Type":"ContainerDied","Data":"bca3af4cc29da6ad162bbb45ce1fa7f4ef2f6d647311987abbeea3864221d083"} Nov 02 18:29:21.112938 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:21.112892 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c" Nov 02 18:29:21.112938 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:21.112901 4471 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bca3af4cc29da6ad162bbb45ce1fa7f4ef2f6d647311987abbeea3864221d083" Nov 02 18:29:21.856990 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:21.856958 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:29:21.857377 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:21.857351 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:29:21.858205 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:21.858183 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:29:28.577612 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:28.577243 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c"] Nov 02 18:29:28.589771 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:28.589730 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae417f29-r9h7c"] Nov 02 18:29:30.093328 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:30.092267 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:29:30.446297 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:30.446213 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d478a85-8b4a-4039-80d9-320a0beb4258" path="/var/lib/kubelet/pods/9d478a85-8b4a-4039-80d9-320a0beb4258/volumes" Nov 02 18:29:31.959271 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:31.959229 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59"] Nov 02 18:29:31.959744 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:31.959488 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" podUID="ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc" containerName="istio-proxy" containerID="cri-o://2746adfdf1544dea5f969617b39d07069d2670d4e342c50298b3bc36e8bcd4bf" gracePeriod=30 Nov 02 18:29:36.860224 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:36.860189 4471 patch_prober.go:29] interesting pod/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:29:36.860670 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:36.860256 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" podUID="ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:29:37.190620 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:37.190586 4471 generic.go:356] "Generic (PLEG): container finished" podID="ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc" containerID="2746adfdf1544dea5f969617b39d07069d2670d4e342c50298b3bc36e8bcd4bf" exitCode=0 Nov 02 18:29:37.190808 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:37.190666 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" event={"ID":"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc","Type":"ContainerDied","Data":"2746adfdf1544dea5f969617b39d07069d2670d4e342c50298b3bc36e8bcd4bf"} Nov 02 18:29:37.190808 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:37.190743 4471 scope.go:122] "RemoveContainer" containerID="2ca205fe00879d12add2d8d6aa20de9427f8b23c6af893a5baf44a5ccad0299f" Nov 02 18:29:37.235472 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:37.235446 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:29:37.320091 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:37.320057 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istio-envoy\") pod \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " Nov 02 18:29:37.320289 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:37.320114 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-credential-socket\") pod \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " Nov 02 18:29:37.320289 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:37.320144 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-workload-certs\") pod \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " Nov 02 18:29:37.320289 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:37.320186 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istio-token\" (UniqueName: \"kubernetes.io/projected/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istio-token\") pod \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " Nov 02 18:29:37.320289 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:37.320205 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-workload-socket\") pod \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " Nov 02 18:29:37.320289 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:37.320264 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istiod-ca-cert\") pod \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " Nov 02 18:29:37.320634 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:37.320381 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-kube-api-access-kzxpx\" (UniqueName: \"kubernetes.io/projected/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-kube-api-access-kzxpx\") pod \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " Nov 02 18:29:37.320634 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:37.320417 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-credential-socket" pod "ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc" (UID: "ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:37.320634 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:37.320429 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istio-podinfo\") pod \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " Nov 02 18:29:37.320634 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:37.320537 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istio-data\") pod \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\" (UID: \"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc\") " Nov 02 18:29:37.320634 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:37.320603 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-workload-socket" pod "ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc" (UID: "ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:37.320911 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:37.320622 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-workload-certs" pod "ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc" (UID: "ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:37.320911 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:37.320821 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istio-data" pod "ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc" (UID: "ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:37.321025 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:37.320997 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istio-data\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:29:37.321025 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:37.321019 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-credential-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:29:37.321148 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:37.321035 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-workload-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:29:37.321148 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:37.321047 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-workload-socket\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:29:37.321406 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:37.321378 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istiod-ca-cert" pod "ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc" (UID: "ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:29:37.322933 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:37.322905 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istio-envoy" pod "ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc" (UID: "ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:37.323032 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:37.323009 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-kube-api-access-kzxpx" pod "ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc" (UID: "ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc"). InnerVolumeSpecName "kube-api-access-kzxpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:37.323277 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:37.323256 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istio-token" pod "ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc" (UID: "ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:37.323374 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:37.323267 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istio-podinfo" pod "ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc" (UID: "ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:29:37.422167 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:37.422078 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzxpx\" (UniqueName: \"kubernetes.io/projected/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-kube-api-access-kzxpx\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:29:37.422167 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:37.422112 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istio-podinfo\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:29:37.422167 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:37.422123 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istio-envoy\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:29:37.422167 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:37.422131 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istio-token\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:29:37.422167 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:37.422138 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc-istiod-ca-cert\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:29:38.196818 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.196790 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" Nov 02 18:29:38.197279 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.196811 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59" event={"ID":"ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc","Type":"ContainerDied","Data":"109e1ab57acabcdda24378572380f9dabc1ffd229b37c8fc72a6e170c2188610"} Nov 02 18:29:38.197279 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.196892 4471 scope.go:122] "RemoveContainer" containerID="2746adfdf1544dea5f969617b39d07069d2670d4e342c50298b3bc36e8bcd4bf" Nov 02 18:29:38.262589 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.262564 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59"] Nov 02 18:29:38.264271 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.264251 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ae417f29-openshift-default-9c4dcc987-xhm59"] Nov 02 18:29:38.439046 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.439014 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc" path="/var/lib/kubelet/pods/ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc/volumes" Nov 02 18:29:38.697989 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.697951 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd"] Nov 02 18:29:38.698415 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.698397 4471 cpu_manager.go:414] "containerMap: removing container" podUID="9d478a85-8b4a-4039-80d9-320a0beb4258" containerName="revoke-keys" Nov 02 18:29:38.698415 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.698414 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d478a85-8b4a-4039-80d9-320a0beb4258" containerName="revoke-keys" Nov 02 18:29:38.698547 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.698429 4471 cpu_manager.go:414] "containerMap: removing container" podUID="ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc" containerName="istio-proxy" Nov 02 18:29:38.698547 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.698434 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc" containerName="istio-proxy" Nov 02 18:29:38.698547 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.698503 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc" containerName="istio-proxy" Nov 02 18:29:38.698547 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.698515 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d478a85-8b4a-4039-80d9-320a0beb4258" containerName="revoke-keys" Nov 02 18:29:38.703452 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.703432 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd" Nov 02 18:29:38.721614 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.721581 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 18:29:38.721614 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.721599 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 18:29:38.724758 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.724729 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd"] Nov 02 18:29:38.732223 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.732199 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-d4ddeb-serving-cert\"" Nov 02 18:29:38.732399 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.732266 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gml5d\"" Nov 02 18:29:38.734040 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.734019 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 18:29:38.835686 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.835652 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjwfx\" (UniqueName: \"kubernetes.io/projected/03542e23-2145-4ab2-8b38-6f0f88d88216-kube-api-access-jjwfx\") pod \"maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd\" (UID: \"03542e23-2145-4ab2-8b38-6f0f88d88216\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd" Nov 02 18:29:38.835868 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.835696 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03542e23-2145-4ab2-8b38-6f0f88d88216-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd\" (UID: \"03542e23-2145-4ab2-8b38-6f0f88d88216\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd" Nov 02 18:29:38.835868 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.835833 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03542e23-2145-4ab2-8b38-6f0f88d88216-metrics-certs\") pod \"maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd\" (UID: \"03542e23-2145-4ab2-8b38-6f0f88d88216\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd" Nov 02 18:29:38.835953 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.835939 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03542e23-2145-4ab2-8b38-6f0f88d88216-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd\" (UID: \"03542e23-2145-4ab2-8b38-6f0f88d88216\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd" Nov 02 18:29:38.936615 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.936588 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03542e23-2145-4ab2-8b38-6f0f88d88216-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd\" (UID: \"03542e23-2145-4ab2-8b38-6f0f88d88216\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd" Nov 02 18:29:38.936763 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.936640 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjwfx\" (UniqueName: \"kubernetes.io/projected/03542e23-2145-4ab2-8b38-6f0f88d88216-kube-api-access-jjwfx\") pod \"maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd\" (UID: \"03542e23-2145-4ab2-8b38-6f0f88d88216\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd" Nov 02 18:29:38.936820 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.936758 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03542e23-2145-4ab2-8b38-6f0f88d88216-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd\" (UID: \"03542e23-2145-4ab2-8b38-6f0f88d88216\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd" Nov 02 18:29:38.936856 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.936819 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03542e23-2145-4ab2-8b38-6f0f88d88216-metrics-certs\") pod \"maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd\" (UID: \"03542e23-2145-4ab2-8b38-6f0f88d88216\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd" Nov 02 18:29:38.937293 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.937265 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03542e23-2145-4ab2-8b38-6f0f88d88216-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd\" (UID: \"03542e23-2145-4ab2-8b38-6f0f88d88216\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd" Nov 02 18:29:38.939094 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.939070 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03542e23-2145-4ab2-8b38-6f0f88d88216-metrics-certs\") pod \"maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd\" (UID: \"03542e23-2145-4ab2-8b38-6f0f88d88216\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd" Nov 02 18:29:38.939193 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.939122 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03542e23-2145-4ab2-8b38-6f0f88d88216-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd\" (UID: \"03542e23-2145-4ab2-8b38-6f0f88d88216\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd" Nov 02 18:29:38.953957 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.953907 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-d4ddeb-d9fc8566c-d7jkz"] Nov 02 18:29:38.954379 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.954364 4471 cpu_manager.go:414] "containerMap: removing container" podUID="ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc" containerName="istio-proxy" Nov 02 18:29:38.954379 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.954379 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc" containerName="istio-proxy" Nov 02 18:29:38.954489 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.954440 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad2ffdb9-fb43-4f2c-a3ea-c0c7492694dc" containerName="istio-proxy" Nov 02 18:29:38.957766 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.957748 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d4ddeb-d9fc8566c-d7jkz" Nov 02 18:29:38.965011 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.964991 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjwfx\" (UniqueName: \"kubernetes.io/projected/03542e23-2145-4ab2-8b38-6f0f88d88216-kube-api-access-jjwfx\") pod \"maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd\" (UID: \"03542e23-2145-4ab2-8b38-6f0f88d88216\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd" Nov 02 18:29:38.977436 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.977396 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-d4ddeb-d9fc8566c-d7jkz"] Nov 02 18:29:38.977636 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.977619 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-d4ddeb\"" Nov 02 18:29:38.978714 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:38.978694 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-d4ddeb-dockercfg-8hvmk\"" Nov 02 18:29:39.042549 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:39.042508 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd" Nov 02 18:29:39.058364 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:39.057569 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-d4ddeb-55c5b488c7-4dwr7"] Nov 02 18:29:39.079355 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:39.070877 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d4ddeb-55c5b488c7-4dwr7" Nov 02 18:29:39.081657 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:39.081625 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-d4ddeb-55c5b488c7-4dwr7"] Nov 02 18:29:39.150605 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:39.150568 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/899e64fc-d451-4ae4-aa5e-17795843a552-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-d4ddeb-d9fc8566c-d7jkz\" (UID: \"899e64fc-d451-4ae4-aa5e-17795843a552\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d4ddeb-d9fc8566c-d7jkz" Nov 02 18:29:39.150743 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:39.150633 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b47bb\" (UniqueName: \"kubernetes.io/projected/899e64fc-d451-4ae4-aa5e-17795843a552-kube-api-access-b47bb\") pod \"payload-pre-processing-e2e-authcfg-http-d4ddeb-d9fc8566c-d7jkz\" (UID: \"899e64fc-d451-4ae4-aa5e-17795843a552\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d4ddeb-d9fc8566c-d7jkz" Nov 02 18:29:39.185387 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:29:39.185354 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:29:39.251519 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:39.251434 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrhgw\" (UniqueName: \"kubernetes.io/projected/a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40-kube-api-access-wrhgw\") pod \"payload-processing-e2e-authcfg-http-d4ddeb-55c5b488c7-4dwr7\" (UID: \"a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-d4ddeb-55c5b488c7-4dwr7" Nov 02 18:29:39.251519 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:39.251483 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/899e64fc-d451-4ae4-aa5e-17795843a552-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-d4ddeb-d9fc8566c-d7jkz\" (UID: \"899e64fc-d451-4ae4-aa5e-17795843a552\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d4ddeb-d9fc8566c-d7jkz" Nov 02 18:29:39.251998 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:39.251525 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b47bb\" (UniqueName: \"kubernetes.io/projected/899e64fc-d451-4ae4-aa5e-17795843a552-kube-api-access-b47bb\") pod \"payload-pre-processing-e2e-authcfg-http-d4ddeb-d9fc8566c-d7jkz\" (UID: \"899e64fc-d451-4ae4-aa5e-17795843a552\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d4ddeb-d9fc8566c-d7jkz" Nov 02 18:29:39.251998 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:39.251659 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-d4ddeb-55c5b488c7-4dwr7\" (UID: \"a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-d4ddeb-55c5b488c7-4dwr7" Nov 02 18:29:39.252240 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:39.252221 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/899e64fc-d451-4ae4-aa5e-17795843a552-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-d4ddeb-d9fc8566c-d7jkz\" (UID: \"899e64fc-d451-4ae4-aa5e-17795843a552\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d4ddeb-d9fc8566c-d7jkz" Nov 02 18:29:39.275138 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:39.275114 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b47bb\" (UniqueName: \"kubernetes.io/projected/899e64fc-d451-4ae4-aa5e-17795843a552-kube-api-access-b47bb\") pod \"payload-pre-processing-e2e-authcfg-http-d4ddeb-d9fc8566c-d7jkz\" (UID: \"899e64fc-d451-4ae4-aa5e-17795843a552\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d4ddeb-d9fc8566c-d7jkz" Nov 02 18:29:39.280944 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:39.280921 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d4ddeb-d9fc8566c-d7jkz" Nov 02 18:29:39.353049 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:39.353009 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrhgw\" (UniqueName: \"kubernetes.io/projected/a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40-kube-api-access-wrhgw\") pod \"payload-processing-e2e-authcfg-http-d4ddeb-55c5b488c7-4dwr7\" (UID: \"a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-d4ddeb-55c5b488c7-4dwr7" Nov 02 18:29:39.353215 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:39.353200 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-d4ddeb-55c5b488c7-4dwr7\" (UID: \"a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-d4ddeb-55c5b488c7-4dwr7" Nov 02 18:29:39.353910 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:39.353885 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-d4ddeb-55c5b488c7-4dwr7\" (UID: \"a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-d4ddeb-55c5b488c7-4dwr7" Nov 02 18:29:39.371461 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:29:39.371438 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03542e23_2145_4ab2_8b38_6f0f88d88216.slice/crio-baa478aa1cb9011be937722d31fa3e8a53838eb34da54be5f712189678e247f1 WatchSource:0}: Error finding container baa478aa1cb9011be937722d31fa3e8a53838eb34da54be5f712189678e247f1: Status 404 returned error can't find the container with id baa478aa1cb9011be937722d31fa3e8a53838eb34da54be5f712189678e247f1 Nov 02 18:29:39.372193 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:39.372161 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd"] Nov 02 18:29:39.372929 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:39.372911 4471 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 18:29:39.376928 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:39.376904 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrhgw\" (UniqueName: \"kubernetes.io/projected/a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40-kube-api-access-wrhgw\") pod \"payload-processing-e2e-authcfg-http-d4ddeb-55c5b488c7-4dwr7\" (UID: \"a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-d4ddeb-55c5b488c7-4dwr7" Nov 02 18:29:39.406454 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:39.406427 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d4ddeb-55c5b488c7-4dwr7" Nov 02 18:29:39.575742 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:39.575667 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-d4ddeb-d9fc8566c-d7jkz"] Nov 02 18:29:39.577061 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:29:39.577027 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod899e64fc_d451_4ae4_aa5e_17795843a552.slice/crio-cda94c1adb47af324ff95087175c3dd8fda64ba0a173e2fda9c60493f5c972b9 WatchSource:0}: Error finding container cda94c1adb47af324ff95087175c3dd8fda64ba0a173e2fda9c60493f5c972b9: Status 404 returned error can't find the container with id cda94c1adb47af324ff95087175c3dd8fda64ba0a173e2fda9c60493f5c972b9 Nov 02 18:29:39.699597 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:39.699570 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-d4ddeb-55c5b488c7-4dwr7"] Nov 02 18:29:39.707518 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:29:39.707489 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c6d6ee_71c5_45c1_b8d2_ffad1f25ce40.slice/crio-c17e126f72850dde6eb38d4d7035e3449d05f6102b4511c1c18badbe2747618c WatchSource:0}: Error finding container c17e126f72850dde6eb38d4d7035e3449d05f6102b4511c1c18badbe2747618c: Status 404 returned error can't find the container with id c17e126f72850dde6eb38d4d7035e3449d05f6102b4511c1c18badbe2747618c Nov 02 18:29:40.209991 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:40.209950 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd" event={"ID":"03542e23-2145-4ab2-8b38-6f0f88d88216","Type":"ContainerStarted","Data":"f41f95c674e6d88e1f965b5226b8c7fed730d5b2d2193d67bc9ab28911963c7c"} Nov 02 18:29:40.210207 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:40.210014 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd" event={"ID":"03542e23-2145-4ab2-8b38-6f0f88d88216","Type":"ContainerStarted","Data":"baa478aa1cb9011be937722d31fa3e8a53838eb34da54be5f712189678e247f1"} Nov 02 18:29:40.210207 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:40.210075 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd" Nov 02 18:29:40.211826 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:40.211799 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d4ddeb-d9fc8566c-d7jkz" event={"ID":"899e64fc-d451-4ae4-aa5e-17795843a552","Type":"ContainerStarted","Data":"d317d2def3c4df7184606b44c8d8db16090a55e4dd98909cb5a66ad5468829a3"} Nov 02 18:29:40.212003 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:40.211849 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d4ddeb-d9fc8566c-d7jkz" event={"ID":"899e64fc-d451-4ae4-aa5e-17795843a552","Type":"ContainerStarted","Data":"cda94c1adb47af324ff95087175c3dd8fda64ba0a173e2fda9c60493f5c972b9"} Nov 02 18:29:40.212003 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:40.211878 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d4ddeb-d9fc8566c-d7jkz" Nov 02 18:29:40.213546 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:40.213521 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d4ddeb-55c5b488c7-4dwr7" event={"ID":"a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40","Type":"ContainerStarted","Data":"95754b8bcf955d6d60319b78962db4e5043548fe07ad0f9d40f67a20da2d930b"} Nov 02 18:29:40.213676 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:40.213559 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d4ddeb-55c5b488c7-4dwr7" event={"ID":"a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40","Type":"ContainerStarted","Data":"c17e126f72850dde6eb38d4d7035e3449d05f6102b4511c1c18badbe2747618c"} Nov 02 18:29:40.213676 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:40.213650 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d4ddeb-55c5b488c7-4dwr7" Nov 02 18:29:40.254542 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:40.254499 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd" podStartSLOduration=1.948579799 podStartE2EDuration="2.254487309s" totalImagesPullingTime="305.90751ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:29:40.253520794 +0000 UTC m=+3597.948876290" watchObservedRunningTime="2026-11-02 18:29:40.254487309 +0000 UTC m=+3597.949842783" Nov 02 18:29:40.295118 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:40.295073 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d4ddeb-55c5b488c7-4dwr7" podStartSLOduration=1.295060626 podStartE2EDuration="1.295060626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:29:40.294872437 +0000 UTC m=+3597.990227912" watchObservedRunningTime="2026-11-02 18:29:40.295060626 +0000 UTC m=+3597.990416098" Nov 02 18:29:40.337842 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:40.337784 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d4ddeb-d9fc8566c-d7jkz" podStartSLOduration=2.3377701379999998 podStartE2EDuration="2.337770138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:29:40.335659257 +0000 UTC m=+3598.031014751" watchObservedRunningTime="2026-11-02 18:29:40.337770138 +0000 UTC m=+3598.033125611" Nov 02 18:29:42.535812 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:42.535780 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:29:46.223928 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:46.223891 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd" Nov 02 18:29:51.220927 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:51.220892 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d4ddeb-55c5b488c7-4dwr7" Nov 02 18:29:51.221858 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:29:51.221841 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d4ddeb-d9fc8566c-d7jkz" Nov 02 18:30:00.153844 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:00.153807 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw"] Nov 02 18:30:00.157499 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:00.157475 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw" Nov 02 18:30:00.187482 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:00.187459 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x9vwr\"" Nov 02 18:30:00.189459 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:00.189437 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw"] Nov 02 18:30:00.293746 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:00.293714 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f77ds\" (UniqueName: \"kubernetes.io/projected/ad731b5a-d7bb-41ef-b97d-72b273873c55-kube-api-access-f77ds\") pod \"maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw\" (UID: \"ad731b5a-d7bb-41ef-b97d-72b273873c55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw" Nov 02 18:30:00.293950 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:00.293833 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad731b5a-d7bb-41ef-b97d-72b273873c55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw\" (UID: \"ad731b5a-d7bb-41ef-b97d-72b273873c55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw" Nov 02 18:30:00.394961 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:00.394931 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad731b5a-d7bb-41ef-b97d-72b273873c55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw\" (UID: \"ad731b5a-d7bb-41ef-b97d-72b273873c55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw" Nov 02 18:30:00.395125 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:00.394999 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f77ds\" (UniqueName: \"kubernetes.io/projected/ad731b5a-d7bb-41ef-b97d-72b273873c55-kube-api-access-f77ds\") pod \"maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw\" (UID: \"ad731b5a-d7bb-41ef-b97d-72b273873c55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw" Nov 02 18:30:00.395606 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:00.395585 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad731b5a-d7bb-41ef-b97d-72b273873c55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw\" (UID: \"ad731b5a-d7bb-41ef-b97d-72b273873c55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw" Nov 02 18:30:00.419223 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:00.419161 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f77ds\" (UniqueName: \"kubernetes.io/projected/ad731b5a-d7bb-41ef-b97d-72b273873c55-kube-api-access-f77ds\") pod \"maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw\" (UID: \"ad731b5a-d7bb-41ef-b97d-72b273873c55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw" Nov 02 18:30:00.470269 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:00.470246 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw" Nov 02 18:30:00.767439 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:30:00.767393 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad731b5a_d7bb_41ef_b97d_72b273873c55.slice/crio-883e6b6574dbb0fc4b04278c02e106a56c5269e4bb86ae94af283c6306749efd WatchSource:0}: Error finding container 883e6b6574dbb0fc4b04278c02e106a56c5269e4bb86ae94af283c6306749efd: Status 404 returned error can't find the container with id 883e6b6574dbb0fc4b04278c02e106a56c5269e4bb86ae94af283c6306749efd Nov 02 18:30:00.767555 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:00.767505 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw"] Nov 02 18:30:01.316355 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:01.316234 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw" event={"ID":"ad731b5a-d7bb-41ef-b97d-72b273873c55","Type":"ContainerStarted","Data":"a592f33173d1dd3f2efb67a2822c6ea44fab020f3a6de22a71f024ae476b2b75"} Nov 02 18:30:01.316355 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:01.316291 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw" event={"ID":"ad731b5a-d7bb-41ef-b97d-72b273873c55","Type":"ContainerStarted","Data":"883e6b6574dbb0fc4b04278c02e106a56c5269e4bb86ae94af283c6306749efd"} Nov 02 18:30:01.357833 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:01.357771 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw" podStartSLOduration=1.357755917 podStartE2EDuration="1.357755917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:01.356699103 +0000 UTC m=+3619.052054604" watchObservedRunningTime="2026-11-02 18:30:01.357755917 +0000 UTC m=+3619.053111389" Nov 02 18:30:02.323486 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:02.323453 4471 generic.go:356] "Generic (PLEG): container finished" podID="ad731b5a-d7bb-41ef-b97d-72b273873c55" containerID="a592f33173d1dd3f2efb67a2822c6ea44fab020f3a6de22a71f024ae476b2b75" exitCode=60 Nov 02 18:30:02.323960 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:02.323545 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw" event={"ID":"ad731b5a-d7bb-41ef-b97d-72b273873c55","Type":"ContainerDied","Data":"a592f33173d1dd3f2efb67a2822c6ea44fab020f3a6de22a71f024ae476b2b75"} Nov 02 18:30:02.323960 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:02.323811 4471 scope.go:122] "RemoveContainer" containerID="a592f33173d1dd3f2efb67a2822c6ea44fab020f3a6de22a71f024ae476b2b75" Nov 02 18:30:03.331067 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:03.331027 4471 generic.go:356] "Generic (PLEG): container finished" podID="ad731b5a-d7bb-41ef-b97d-72b273873c55" containerID="f49470767b6575ca428855c4617c4fc852bb36a7389014d35951c449fa9f1059" exitCode=60 Nov 02 18:30:03.331542 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:03.331084 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw" event={"ID":"ad731b5a-d7bb-41ef-b97d-72b273873c55","Type":"ContainerDied","Data":"f49470767b6575ca428855c4617c4fc852bb36a7389014d35951c449fa9f1059"} Nov 02 18:30:03.331542 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:03.331145 4471 scope.go:122] "RemoveContainer" containerID="a592f33173d1dd3f2efb67a2822c6ea44fab020f3a6de22a71f024ae476b2b75" Nov 02 18:30:03.331542 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:03.331390 4471 scope.go:122] "RemoveContainer" containerID="f49470767b6575ca428855c4617c4fc852bb36a7389014d35951c449fa9f1059" Nov 02 18:30:03.331684 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:30:03.331661 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw_odh-ai-gateway-infra(ad731b5a-d7bb-41ef-b97d-72b273873c55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw" podUID="ad731b5a-d7bb-41ef-b97d-72b273873c55" Nov 02 18:30:04.337647 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:04.337617 4471 scope.go:122] "RemoveContainer" containerID="f49470767b6575ca428855c4617c4fc852bb36a7389014d35951c449fa9f1059" Nov 02 18:30:04.338105 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:30:04.337897 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw_odh-ai-gateway-infra(ad731b5a-d7bb-41ef-b97d-72b273873c55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw" podUID="ad731b5a-d7bb-41ef-b97d-72b273873c55" Nov 02 18:30:09.186040 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:30:09.185989 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:30:17.434002 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:17.433974 4471 scope.go:122] "RemoveContainer" containerID="f49470767b6575ca428855c4617c4fc852bb36a7389014d35951c449fa9f1059" Nov 02 18:30:18.424051 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:18.424016 4471 generic.go:356] "Generic (PLEG): container finished" podID="ad731b5a-d7bb-41ef-b97d-72b273873c55" containerID="f49c89705b42a0e71fd91d2ae69dc0bcc72d3b11c8395b4485265a4553469ab7" exitCode=60 Nov 02 18:30:18.424259 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:18.424069 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw" event={"ID":"ad731b5a-d7bb-41ef-b97d-72b273873c55","Type":"ContainerDied","Data":"f49c89705b42a0e71fd91d2ae69dc0bcc72d3b11c8395b4485265a4553469ab7"} Nov 02 18:30:18.424259 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:18.424118 4471 scope.go:122] "RemoveContainer" containerID="f49470767b6575ca428855c4617c4fc852bb36a7389014d35951c449fa9f1059" Nov 02 18:30:18.424565 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:18.424544 4471 scope.go:122] "RemoveContainer" containerID="f49c89705b42a0e71fd91d2ae69dc0bcc72d3b11c8395b4485265a4553469ab7" Nov 02 18:30:18.424832 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:30:18.424808 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw_odh-ai-gateway-infra(ad731b5a-d7bb-41ef-b97d-72b273873c55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw" podUID="ad731b5a-d7bb-41ef-b97d-72b273873c55" Nov 02 18:30:19.477050 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:19.477007 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw"] Nov 02 18:30:19.620652 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:19.620624 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw" Nov 02 18:30:19.697370 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:19.697300 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad731b5a-d7bb-41ef-b97d-72b273873c55-kube-api-access-f77ds\" (UniqueName: \"kubernetes.io/projected/ad731b5a-d7bb-41ef-b97d-72b273873c55-kube-api-access-f77ds\") pod \"ad731b5a-d7bb-41ef-b97d-72b273873c55\" (UID: \"ad731b5a-d7bb-41ef-b97d-72b273873c55\") " Nov 02 18:30:19.697574 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:19.697435 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad731b5a-d7bb-41ef-b97d-72b273873c55-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad731b5a-d7bb-41ef-b97d-72b273873c55-maas-api-service-ca\") pod \"ad731b5a-d7bb-41ef-b97d-72b273873c55\" (UID: \"ad731b5a-d7bb-41ef-b97d-72b273873c55\") " Nov 02 18:30:19.698075 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:19.698045 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad731b5a-d7bb-41ef-b97d-72b273873c55-maas-api-service-ca" pod "ad731b5a-d7bb-41ef-b97d-72b273873c55" (UID: "ad731b5a-d7bb-41ef-b97d-72b273873c55"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:30:19.699785 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:19.699759 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad731b5a-d7bb-41ef-b97d-72b273873c55-kube-api-access-f77ds" pod "ad731b5a-d7bb-41ef-b97d-72b273873c55" (UID: "ad731b5a-d7bb-41ef-b97d-72b273873c55"). InnerVolumeSpecName "kube-api-access-f77ds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:19.798461 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:19.798426 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f77ds\" (UniqueName: \"kubernetes.io/projected/ad731b5a-d7bb-41ef-b97d-72b273873c55-kube-api-access-f77ds\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:30:19.798461 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:19.798456 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad731b5a-d7bb-41ef-b97d-72b273873c55-maas-api-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:30:20.445769 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:20.445735 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw" Nov 02 18:30:20.446457 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:20.446426 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw" event={"ID":"ad731b5a-d7bb-41ef-b97d-72b273873c55","Type":"ContainerDied","Data":"883e6b6574dbb0fc4b04278c02e106a56c5269e4bb86ae94af283c6306749efd"} Nov 02 18:30:20.446606 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:20.446480 4471 scope.go:122] "RemoveContainer" containerID="f49c89705b42a0e71fd91d2ae69dc0bcc72d3b11c8395b4485265a4553469ab7" Nov 02 18:30:20.498745 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:20.498715 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw"] Nov 02 18:30:20.511581 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:20.511556 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-d4ddeb-29894070-kzcfw"] Nov 02 18:30:22.439628 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:30:22.439594 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad731b5a-d7bb-41ef-b97d-72b273873c55" path="/var/lib/kubelet/pods/ad731b5a-d7bb-41ef-b97d-72b273873c55/volumes" Nov 02 18:30:39.183393 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:30:39.183352 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:31:09.183504 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:31:09.183462 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:31:12.156517 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:12.156475 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d4ddeb-gr2b8"] Nov 02 18:31:12.157123 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:12.157104 4471 cpu_manager.go:414] "containerMap: removing container" podUID="ad731b5a-d7bb-41ef-b97d-72b273873c55" containerName="cleanup" Nov 02 18:31:12.157177 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:12.157126 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad731b5a-d7bb-41ef-b97d-72b273873c55" containerName="cleanup" Nov 02 18:31:12.157177 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:12.157146 4471 cpu_manager.go:414] "containerMap: removing container" podUID="ad731b5a-d7bb-41ef-b97d-72b273873c55" containerName="cleanup" Nov 02 18:31:12.157177 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:12.157155 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad731b5a-d7bb-41ef-b97d-72b273873c55" containerName="cleanup" Nov 02 18:31:12.157177 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:12.157171 4471 cpu_manager.go:414] "containerMap: removing container" podUID="ad731b5a-d7bb-41ef-b97d-72b273873c55" containerName="cleanup" Nov 02 18:31:12.157434 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:12.157179 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad731b5a-d7bb-41ef-b97d-72b273873c55" containerName="cleanup" Nov 02 18:31:12.157434 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:12.157276 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad731b5a-d7bb-41ef-b97d-72b273873c55" containerName="cleanup" Nov 02 18:31:12.157434 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:12.157287 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad731b5a-d7bb-41ef-b97d-72b273873c55" containerName="cleanup" Nov 02 18:31:12.157434 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:12.157298 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad731b5a-d7bb-41ef-b97d-72b273873c55" containerName="cleanup" Nov 02 18:31:12.160784 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:12.160766 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d4ddeb-gr2b8" Nov 02 18:31:12.177467 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:12.177437 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:31:12.179835 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:12.179807 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d4ddeb-gr2b8"] Nov 02 18:31:12.190047 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:12.190021 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x9vwr\"" Nov 02 18:31:12.252923 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:12.252885 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d410369b-6fff-4617-9efd-7c2690a91fc5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d4ddeb-gr2b8\" (UID: \"d410369b-6fff-4617-9efd-7c2690a91fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d4ddeb-gr2b8" Nov 02 18:31:12.252923 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:12.252928 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-552c9\" (UniqueName: \"kubernetes.io/projected/d410369b-6fff-4617-9efd-7c2690a91fc5-kube-api-access-552c9\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d4ddeb-gr2b8\" (UID: \"d410369b-6fff-4617-9efd-7c2690a91fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d4ddeb-gr2b8" Nov 02 18:31:12.253200 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:12.252955 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d410369b-6fff-4617-9efd-7c2690a91fc5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d4ddeb-gr2b8\" (UID: \"d410369b-6fff-4617-9efd-7c2690a91fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d4ddeb-gr2b8" Nov 02 18:31:12.354263 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:12.354220 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d410369b-6fff-4617-9efd-7c2690a91fc5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d4ddeb-gr2b8\" (UID: \"d410369b-6fff-4617-9efd-7c2690a91fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d4ddeb-gr2b8" Nov 02 18:31:12.354263 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:12.354262 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-552c9\" (UniqueName: \"kubernetes.io/projected/d410369b-6fff-4617-9efd-7c2690a91fc5-kube-api-access-552c9\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d4ddeb-gr2b8\" (UID: \"d410369b-6fff-4617-9efd-7c2690a91fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d4ddeb-gr2b8" Nov 02 18:31:12.354498 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:12.354294 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d410369b-6fff-4617-9efd-7c2690a91fc5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d4ddeb-gr2b8\" (UID: \"d410369b-6fff-4617-9efd-7c2690a91fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d4ddeb-gr2b8" Nov 02 18:31:12.355015 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:12.354989 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d410369b-6fff-4617-9efd-7c2690a91fc5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d4ddeb-gr2b8\" (UID: \"d410369b-6fff-4617-9efd-7c2690a91fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d4ddeb-gr2b8" Nov 02 18:31:12.356649 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:12.356622 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d410369b-6fff-4617-9efd-7c2690a91fc5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d4ddeb-gr2b8\" (UID: \"d410369b-6fff-4617-9efd-7c2690a91fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d4ddeb-gr2b8" Nov 02 18:31:12.379058 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:12.379025 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-552c9\" (UniqueName: \"kubernetes.io/projected/d410369b-6fff-4617-9efd-7c2690a91fc5-kube-api-access-552c9\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d4ddeb-gr2b8\" (UID: \"d410369b-6fff-4617-9efd-7c2690a91fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d4ddeb-gr2b8" Nov 02 18:31:12.473549 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:12.473455 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d4ddeb-gr2b8" Nov 02 18:31:12.771094 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:12.771068 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d4ddeb-gr2b8"] Nov 02 18:31:12.771233 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:31:12.771208 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd410369b_6fff_4617_9efd_7c2690a91fc5.slice/crio-5d82ef814e490900022afee734ab1c791a76b7e7ce9462acb459496b0e4b7ec2 WatchSource:0}: Error finding container 5d82ef814e490900022afee734ab1c791a76b7e7ce9462acb459496b0e4b7ec2: Status 404 returned error can't find the container with id 5d82ef814e490900022afee734ab1c791a76b7e7ce9462acb459496b0e4b7ec2 Nov 02 18:31:13.710482 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:13.710449 4471 generic.go:356] "Generic (PLEG): container finished" podID="d410369b-6fff-4617-9efd-7c2690a91fc5" containerID="47988c4c472d036bb1cdb8bf56428fa23fe80079f49b4fe6cc5e6e8c6b3fb4a2" exitCode=0 Nov 02 18:31:13.710882 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:13.710499 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d4ddeb-gr2b8" event={"ID":"d410369b-6fff-4617-9efd-7c2690a91fc5","Type":"ContainerDied","Data":"47988c4c472d036bb1cdb8bf56428fa23fe80079f49b4fe6cc5e6e8c6b3fb4a2"} Nov 02 18:31:13.710882 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:13.710541 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d4ddeb-gr2b8" event={"ID":"d410369b-6fff-4617-9efd-7c2690a91fc5","Type":"ContainerStarted","Data":"5d82ef814e490900022afee734ab1c791a76b7e7ce9462acb459496b0e4b7ec2"} Nov 02 18:31:14.846228 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:14.846203 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d4ddeb-gr2b8" Nov 02 18:31:14.981481 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:14.981390 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d410369b-6fff-4617-9efd-7c2690a91fc5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d410369b-6fff-4617-9efd-7c2690a91fc5-maas-api-xks-ca\") pod \"d410369b-6fff-4617-9efd-7c2690a91fc5\" (UID: \"d410369b-6fff-4617-9efd-7c2690a91fc5\") " Nov 02 18:31:14.981481 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:14.981439 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d410369b-6fff-4617-9efd-7c2690a91fc5-kube-api-access-552c9\" (UniqueName: \"kubernetes.io/projected/d410369b-6fff-4617-9efd-7c2690a91fc5-kube-api-access-552c9\") pod \"d410369b-6fff-4617-9efd-7c2690a91fc5\" (UID: \"d410369b-6fff-4617-9efd-7c2690a91fc5\") " Nov 02 18:31:14.981481 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:14.981459 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d410369b-6fff-4617-9efd-7c2690a91fc5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d410369b-6fff-4617-9efd-7c2690a91fc5-maas-api-openshift-ca\") pod \"d410369b-6fff-4617-9efd-7c2690a91fc5\" (UID: \"d410369b-6fff-4617-9efd-7c2690a91fc5\") " Nov 02 18:31:14.981879 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:14.981854 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d410369b-6fff-4617-9efd-7c2690a91fc5-maas-api-openshift-ca" pod "d410369b-6fff-4617-9efd-7c2690a91fc5" (UID: "d410369b-6fff-4617-9efd-7c2690a91fc5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:31:14.983616 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:14.983592 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d410369b-6fff-4617-9efd-7c2690a91fc5-maas-api-xks-ca" pod "d410369b-6fff-4617-9efd-7c2690a91fc5" (UID: "d410369b-6fff-4617-9efd-7c2690a91fc5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:31:14.983693 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:14.983596 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d410369b-6fff-4617-9efd-7c2690a91fc5-kube-api-access-552c9" pod "d410369b-6fff-4617-9efd-7c2690a91fc5" (UID: "d410369b-6fff-4617-9efd-7c2690a91fc5"). InnerVolumeSpecName "kube-api-access-552c9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:31:15.082932 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:15.082895 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d410369b-6fff-4617-9efd-7c2690a91fc5-maas-api-xks-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:31:15.082932 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:15.082928 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-552c9\" (UniqueName: \"kubernetes.io/projected/d410369b-6fff-4617-9efd-7c2690a91fc5-kube-api-access-552c9\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:31:15.083150 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:15.082941 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d410369b-6fff-4617-9efd-7c2690a91fc5-maas-api-openshift-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:31:15.720338 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:15.720289 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d4ddeb-gr2b8" Nov 02 18:31:15.720338 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:15.720329 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d4ddeb-gr2b8" event={"ID":"d410369b-6fff-4617-9efd-7c2690a91fc5","Type":"ContainerDied","Data":"5d82ef814e490900022afee734ab1c791a76b7e7ce9462acb459496b0e4b7ec2"} Nov 02 18:31:15.720565 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:15.720377 4471 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d82ef814e490900022afee734ab1c791a76b7e7ce9462acb459496b0e4b7ec2" Nov 02 18:31:22.682722 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:22.682684 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-d4ddeb-55c5b488c7-4dwr7"] Nov 02 18:31:22.717096 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:22.717050 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-d4ddeb-d9fc8566c-d7jkz"] Nov 02 18:31:27.684022 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:27.683978 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d4ddeb-55c5b488c7-4dwr7" podUID="a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40" containerName="payload-processing" containerID="cri-o://95754b8bcf955d6d60319b78962db4e5043548fe07ad0f9d40f67a20da2d930b" gracePeriod=25 Nov 02 18:31:27.717802 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:27.717766 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d4ddeb-d9fc8566c-d7jkz" podUID="899e64fc-d451-4ae4-aa5e-17795843a552" containerName="payload-pre-processing" containerID="cri-o://d317d2def3c4df7184606b44c8d8db16090a55e4dd98909cb5a66ad5468829a3" gracePeriod=25 Nov 02 18:31:27.972603 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:27.972573 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d4ddeb-55c5b488c7-4dwr7" Nov 02 18:31:27.998587 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:27.998563 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d4ddeb-d9fc8566c-d7jkz" Nov 02 18:31:28.013382 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:28.013350 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40-plugins-config-volume\") pod \"a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40\" (UID: \"a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40\") " Nov 02 18:31:28.013382 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:28.013453 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/899e64fc-d451-4ae4-aa5e-17795843a552-kube-api-access-b47bb\" (UniqueName: \"kubernetes.io/projected/899e64fc-d451-4ae4-aa5e-17795843a552-kube-api-access-b47bb\") pod \"899e64fc-d451-4ae4-aa5e-17795843a552\" (UID: \"899e64fc-d451-4ae4-aa5e-17795843a552\") " Nov 02 18:31:28.013753 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:28.013489 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/899e64fc-d451-4ae4-aa5e-17795843a552-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/899e64fc-d451-4ae4-aa5e-17795843a552-plugins-config-volume\") pod \"899e64fc-d451-4ae4-aa5e-17795843a552\" (UID: \"899e64fc-d451-4ae4-aa5e-17795843a552\") " Nov 02 18:31:28.013753 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:28.013637 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40-kube-api-access-wrhgw\" (UniqueName: \"kubernetes.io/projected/a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40-kube-api-access-wrhgw\") pod \"a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40\" (UID: \"a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40\") " Nov 02 18:31:28.014286 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:28.014243 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/899e64fc-d451-4ae4-aa5e-17795843a552-plugins-config-volume" pod "899e64fc-d451-4ae4-aa5e-17795843a552" (UID: "899e64fc-d451-4ae4-aa5e-17795843a552"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:31:28.015253 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:28.015196 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40-plugins-config-volume" pod "a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40" (UID: "a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:31:28.019586 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:28.019560 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40-kube-api-access-wrhgw" pod "a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40" (UID: "a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40"). InnerVolumeSpecName "kube-api-access-wrhgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:31:28.021809 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:28.021777 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/899e64fc-d451-4ae4-aa5e-17795843a552-kube-api-access-b47bb" pod "899e64fc-d451-4ae4-aa5e-17795843a552" (UID: "899e64fc-d451-4ae4-aa5e-17795843a552"). InnerVolumeSpecName "kube-api-access-b47bb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:31:28.114594 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:28.114558 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b47bb\" (UniqueName: \"kubernetes.io/projected/899e64fc-d451-4ae4-aa5e-17795843a552-kube-api-access-b47bb\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:31:28.114594 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:28.114585 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/899e64fc-d451-4ae4-aa5e-17795843a552-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:31:28.114594 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:28.114595 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrhgw\" (UniqueName: \"kubernetes.io/projected/a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40-kube-api-access-wrhgw\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:31:28.114594 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:28.114605 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40-plugins-config-volume\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:31:28.782665 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:28.782622 4471 generic.go:356] "Generic (PLEG): container finished" podID="a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40" containerID="95754b8bcf955d6d60319b78962db4e5043548fe07ad0f9d40f67a20da2d930b" exitCode=0 Nov 02 18:31:28.783121 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:28.782676 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d4ddeb-55c5b488c7-4dwr7" Nov 02 18:31:28.783121 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:28.782702 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d4ddeb-55c5b488c7-4dwr7" event={"ID":"a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40","Type":"ContainerDied","Data":"95754b8bcf955d6d60319b78962db4e5043548fe07ad0f9d40f67a20da2d930b"} Nov 02 18:31:28.783121 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:28.782755 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d4ddeb-55c5b488c7-4dwr7" event={"ID":"a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40","Type":"ContainerDied","Data":"c17e126f72850dde6eb38d4d7035e3449d05f6102b4511c1c18badbe2747618c"} Nov 02 18:31:28.783121 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:28.782776 4471 scope.go:122] "RemoveContainer" containerID="95754b8bcf955d6d60319b78962db4e5043548fe07ad0f9d40f67a20da2d930b" Nov 02 18:31:28.784272 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:28.784250 4471 generic.go:356] "Generic (PLEG): container finished" podID="899e64fc-d451-4ae4-aa5e-17795843a552" containerID="d317d2def3c4df7184606b44c8d8db16090a55e4dd98909cb5a66ad5468829a3" exitCode=0 Nov 02 18:31:28.784430 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:28.784279 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d4ddeb-d9fc8566c-d7jkz" event={"ID":"899e64fc-d451-4ae4-aa5e-17795843a552","Type":"ContainerDied","Data":"d317d2def3c4df7184606b44c8d8db16090a55e4dd98909cb5a66ad5468829a3"} Nov 02 18:31:28.784430 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:28.784335 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d4ddeb-d9fc8566c-d7jkz" event={"ID":"899e64fc-d451-4ae4-aa5e-17795843a552","Type":"ContainerDied","Data":"cda94c1adb47af324ff95087175c3dd8fda64ba0a173e2fda9c60493f5c972b9"} Nov 02 18:31:28.784430 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:28.784343 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d4ddeb-d9fc8566c-d7jkz" Nov 02 18:31:28.791827 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:28.791809 4471 scope.go:122] "RemoveContainer" containerID="95754b8bcf955d6d60319b78962db4e5043548fe07ad0f9d40f67a20da2d930b" Nov 02 18:31:28.792079 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:31:28.792059 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95754b8bcf955d6d60319b78962db4e5043548fe07ad0f9d40f67a20da2d930b\": container with ID starting with 95754b8bcf955d6d60319b78962db4e5043548fe07ad0f9d40f67a20da2d930b not found: ID does not exist" containerID="95754b8bcf955d6d60319b78962db4e5043548fe07ad0f9d40f67a20da2d930b" Nov 02 18:31:28.792125 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:28.792090 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95754b8bcf955d6d60319b78962db4e5043548fe07ad0f9d40f67a20da2d930b"} err="failed to get container status \"95754b8bcf955d6d60319b78962db4e5043548fe07ad0f9d40f67a20da2d930b\": rpc error: code = NotFound desc = could not find container \"95754b8bcf955d6d60319b78962db4e5043548fe07ad0f9d40f67a20da2d930b\": container with ID starting with 95754b8bcf955d6d60319b78962db4e5043548fe07ad0f9d40f67a20da2d930b not found: ID does not exist" Nov 02 18:31:28.792125 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:28.792115 4471 scope.go:122] "RemoveContainer" containerID="d317d2def3c4df7184606b44c8d8db16090a55e4dd98909cb5a66ad5468829a3" Nov 02 18:31:28.801987 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:28.801970 4471 scope.go:122] "RemoveContainer" containerID="d317d2def3c4df7184606b44c8d8db16090a55e4dd98909cb5a66ad5468829a3" Nov 02 18:31:28.802300 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:31:28.802279 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d317d2def3c4df7184606b44c8d8db16090a55e4dd98909cb5a66ad5468829a3\": container with ID starting with d317d2def3c4df7184606b44c8d8db16090a55e4dd98909cb5a66ad5468829a3 not found: ID does not exist" containerID="d317d2def3c4df7184606b44c8d8db16090a55e4dd98909cb5a66ad5468829a3" Nov 02 18:31:28.802388 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:28.802333 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d317d2def3c4df7184606b44c8d8db16090a55e4dd98909cb5a66ad5468829a3"} err="failed to get container status \"d317d2def3c4df7184606b44c8d8db16090a55e4dd98909cb5a66ad5468829a3\": rpc error: code = NotFound desc = could not find container \"d317d2def3c4df7184606b44c8d8db16090a55e4dd98909cb5a66ad5468829a3\": container with ID starting with d317d2def3c4df7184606b44c8d8db16090a55e4dd98909cb5a66ad5468829a3 not found: ID does not exist" Nov 02 18:31:28.837568 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:28.837541 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-d4ddeb-55c5b488c7-4dwr7"] Nov 02 18:31:28.850109 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:28.850078 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-d4ddeb-55c5b488c7-4dwr7"] Nov 02 18:31:28.875895 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:28.875866 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-d4ddeb-d9fc8566c-d7jkz"] Nov 02 18:31:28.879983 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:28.879958 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-d4ddeb-d9fc8566c-d7jkz"] Nov 02 18:31:29.384446 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:29.384410 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd"] Nov 02 18:31:29.384630 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:29.384606 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd" podUID="03542e23-2145-4ab2-8b38-6f0f88d88216" containerName="maas-api" containerID="cri-o://f41f95c674e6d88e1f965b5226b8c7fed730d5b2d2193d67bc9ab28911963c7c" gracePeriod=30 Nov 02 18:31:29.662012 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:29.661922 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd" Nov 02 18:31:29.759411 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:29.759376 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03542e23-2145-4ab2-8b38-6f0f88d88216-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03542e23-2145-4ab2-8b38-6f0f88d88216-maas-api-tls\") pod \"03542e23-2145-4ab2-8b38-6f0f88d88216\" (UID: \"03542e23-2145-4ab2-8b38-6f0f88d88216\") " Nov 02 18:31:29.759568 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:29.759484 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03542e23-2145-4ab2-8b38-6f0f88d88216-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03542e23-2145-4ab2-8b38-6f0f88d88216-openshift-service-ca\") pod \"03542e23-2145-4ab2-8b38-6f0f88d88216\" (UID: \"03542e23-2145-4ab2-8b38-6f0f88d88216\") " Nov 02 18:31:29.759568 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:29.759524 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03542e23-2145-4ab2-8b38-6f0f88d88216-metrics-certs\" (UniqueName: \"kubernetes.io/secret/03542e23-2145-4ab2-8b38-6f0f88d88216-metrics-certs\") pod \"03542e23-2145-4ab2-8b38-6f0f88d88216\" (UID: \"03542e23-2145-4ab2-8b38-6f0f88d88216\") " Nov 02 18:31:29.759675 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:29.759579 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03542e23-2145-4ab2-8b38-6f0f88d88216-kube-api-access-jjwfx\" (UniqueName: \"kubernetes.io/projected/03542e23-2145-4ab2-8b38-6f0f88d88216-kube-api-access-jjwfx\") pod \"03542e23-2145-4ab2-8b38-6f0f88d88216\" (UID: \"03542e23-2145-4ab2-8b38-6f0f88d88216\") " Nov 02 18:31:29.759838 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:29.759814 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03542e23-2145-4ab2-8b38-6f0f88d88216-openshift-service-ca" pod "03542e23-2145-4ab2-8b38-6f0f88d88216" (UID: "03542e23-2145-4ab2-8b38-6f0f88d88216"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:31:29.759948 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:29.759906 4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03542e23-2145-4ab2-8b38-6f0f88d88216-openshift-service-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:31:29.761594 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:29.761567 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03542e23-2145-4ab2-8b38-6f0f88d88216-maas-api-tls" pod "03542e23-2145-4ab2-8b38-6f0f88d88216" (UID: "03542e23-2145-4ab2-8b38-6f0f88d88216"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:31:29.761594 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:29.761585 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03542e23-2145-4ab2-8b38-6f0f88d88216-metrics-certs" pod "03542e23-2145-4ab2-8b38-6f0f88d88216" (UID: "03542e23-2145-4ab2-8b38-6f0f88d88216"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:31:29.761761 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:29.761666 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03542e23-2145-4ab2-8b38-6f0f88d88216-kube-api-access-jjwfx" pod "03542e23-2145-4ab2-8b38-6f0f88d88216" (UID: "03542e23-2145-4ab2-8b38-6f0f88d88216"). InnerVolumeSpecName "kube-api-access-jjwfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:31:29.790750 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:29.790719 4471 generic.go:356] "Generic (PLEG): container finished" podID="03542e23-2145-4ab2-8b38-6f0f88d88216" containerID="f41f95c674e6d88e1f965b5226b8c7fed730d5b2d2193d67bc9ab28911963c7c" exitCode=0 Nov 02 18:31:29.791173 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:29.790779 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd" Nov 02 18:31:29.791173 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:29.790807 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd" event={"ID":"03542e23-2145-4ab2-8b38-6f0f88d88216","Type":"ContainerDied","Data":"f41f95c674e6d88e1f965b5226b8c7fed730d5b2d2193d67bc9ab28911963c7c"} Nov 02 18:31:29.791173 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:29.790867 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd" event={"ID":"03542e23-2145-4ab2-8b38-6f0f88d88216","Type":"ContainerDied","Data":"baa478aa1cb9011be937722d31fa3e8a53838eb34da54be5f712189678e247f1"} Nov 02 18:31:29.791173 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:29.790916 4471 scope.go:122] "RemoveContainer" containerID="f41f95c674e6d88e1f965b5226b8c7fed730d5b2d2193d67bc9ab28911963c7c" Nov 02 18:31:29.800023 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:29.800007 4471 scope.go:122] "RemoveContainer" containerID="f41f95c674e6d88e1f965b5226b8c7fed730d5b2d2193d67bc9ab28911963c7c" Nov 02 18:31:29.800288 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:31:29.800270 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f41f95c674e6d88e1f965b5226b8c7fed730d5b2d2193d67bc9ab28911963c7c\": container with ID starting with f41f95c674e6d88e1f965b5226b8c7fed730d5b2d2193d67bc9ab28911963c7c not found: ID does not exist" containerID="f41f95c674e6d88e1f965b5226b8c7fed730d5b2d2193d67bc9ab28911963c7c" Nov 02 18:31:29.800409 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:29.800295 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f41f95c674e6d88e1f965b5226b8c7fed730d5b2d2193d67bc9ab28911963c7c"} err="failed to get container status \"f41f95c674e6d88e1f965b5226b8c7fed730d5b2d2193d67bc9ab28911963c7c\": rpc error: code = NotFound desc = could not find container \"f41f95c674e6d88e1f965b5226b8c7fed730d5b2d2193d67bc9ab28911963c7c\": container with ID starting with f41f95c674e6d88e1f965b5226b8c7fed730d5b2d2193d67bc9ab28911963c7c not found: ID does not exist" Nov 02 18:31:29.869021 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:29.868955 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd"] Nov 02 18:31:29.869021 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:29.868978 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjwfx\" (UniqueName: \"kubernetes.io/projected/03542e23-2145-4ab2-8b38-6f0f88d88216-kube-api-access-jjwfx\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:31:29.869294 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:29.869040 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03542e23-2145-4ab2-8b38-6f0f88d88216-maas-api-tls\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:31:29.869294 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:29.869054 4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03542e23-2145-4ab2-8b38-6f0f88d88216-metrics-certs\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:31:29.871273 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:29.871242 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d4ddeb-86c8584d75-wqszd"] Nov 02 18:31:30.193405 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:30.193370 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d4ddeb-gr2b8"] Nov 02 18:31:30.195988 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:30.195960 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d4ddeb-gr2b8"] Nov 02 18:31:30.439127 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:30.439090 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03542e23-2145-4ab2-8b38-6f0f88d88216" path="/var/lib/kubelet/pods/03542e23-2145-4ab2-8b38-6f0f88d88216/volumes" Nov 02 18:31:30.439564 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:30.439549 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="899e64fc-d451-4ae4-aa5e-17795843a552" path="/var/lib/kubelet/pods/899e64fc-d451-4ae4-aa5e-17795843a552/volumes" Nov 02 18:31:30.439925 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:30.439912 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40" path="/var/lib/kubelet/pods/a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40/volumes" Nov 02 18:31:30.440671 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:30.440654 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d410369b-6fff-4617-9efd-7c2690a91fc5" path="/var/lib/kubelet/pods/d410369b-6fff-4617-9efd-7c2690a91fc5/volumes" Nov 02 18:31:32.676624 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:32.676579 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:31:39.182267 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:31:39.182230 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:31:40.063579 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:40.063543 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:31:42.848329 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:42.848160 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5hhwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:31:42.848329 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:42.848246 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5hhwn" podUID="a482b42d-8782-433f-a05f-391d4516a850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:31:42.892337 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:42.891400 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-f6kf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:31:42.892337 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:42.891461 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6kf2" podUID="c40d5295-d354-4be9-89b8-6bd96ae2e056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:31:48.251416 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:48.251384 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:31:54.895828 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:54.895746 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7tgj6"] Nov 02 18:31:54.896278 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:54.896214 4471 cpu_manager.go:414] "containerMap: removing container" podUID="a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40" containerName="payload-processing" Nov 02 18:31:54.896278 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:54.896228 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40" containerName="payload-processing" Nov 02 18:31:54.896278 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:54.896247 4471 cpu_manager.go:414] "containerMap: removing container" podUID="d410369b-6fff-4617-9efd-7c2690a91fc5" containerName="revoke-keys" Nov 02 18:31:54.896278 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:54.896252 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d410369b-6fff-4617-9efd-7c2690a91fc5" containerName="revoke-keys" Nov 02 18:31:54.896278 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:54.896264 4471 cpu_manager.go:414] "containerMap: removing container" podUID="03542e23-2145-4ab2-8b38-6f0f88d88216" containerName="maas-api" Nov 02 18:31:54.896278 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:54.896269 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03542e23-2145-4ab2-8b38-6f0f88d88216" containerName="maas-api" Nov 02 18:31:54.896539 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:54.896285 4471 cpu_manager.go:414] "containerMap: removing container" podUID="899e64fc-d451-4ae4-aa5e-17795843a552" containerName="payload-pre-processing" Nov 02 18:31:54.896539 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:54.896290 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="899e64fc-d451-4ae4-aa5e-17795843a552" containerName="payload-pre-processing" Nov 02 18:31:54.896539 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:54.896376 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="d410369b-6fff-4617-9efd-7c2690a91fc5" containerName="revoke-keys" Nov 02 18:31:54.896539 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:54.896387 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="03542e23-2145-4ab2-8b38-6f0f88d88216" containerName="maas-api" Nov 02 18:31:54.896539 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:54.896395 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="899e64fc-d451-4ae4-aa5e-17795843a552" containerName="payload-pre-processing" Nov 02 18:31:54.896539 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:54.896403 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8c6d6ee-71c5-45c1-b8d2-ffad1f25ce40" containerName="payload-processing" Nov 02 18:31:54.899613 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:54.899586 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7tgj6" Nov 02 18:31:54.914976 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:54.914953 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:31:54.927765 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:54.927698 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 18:31:54.927765 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:54.927701 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 18:31:54.927765 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:54.927719 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x9vwr\"" Nov 02 18:31:54.930043 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:54.930016 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7tgj6"] Nov 02 18:31:54.988598 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:54.988561 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd170349-0424-48fa-a5e5-24781e3a21bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-7tgj6\" (UID: \"fd170349-0424-48fa-a5e5-24781e3a21bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7tgj6" Nov 02 18:31:54.988762 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:54.988611 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd170349-0424-48fa-a5e5-24781e3a21bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-7tgj6\" (UID: \"fd170349-0424-48fa-a5e5-24781e3a21bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7tgj6" Nov 02 18:31:54.988762 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:54.988728 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jqw2\" (UniqueName: \"kubernetes.io/projected/fd170349-0424-48fa-a5e5-24781e3a21bb-kube-api-access-2jqw2\") pod \"maas-api-revoke-sub-f03629db-7tgj6\" (UID: \"fd170349-0424-48fa-a5e5-24781e3a21bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7tgj6" Nov 02 18:31:55.089389 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:55.089349 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd170349-0424-48fa-a5e5-24781e3a21bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-7tgj6\" (UID: \"fd170349-0424-48fa-a5e5-24781e3a21bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7tgj6" Nov 02 18:31:55.089567 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:55.089400 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jqw2\" (UniqueName: \"kubernetes.io/projected/fd170349-0424-48fa-a5e5-24781e3a21bb-kube-api-access-2jqw2\") pod \"maas-api-revoke-sub-f03629db-7tgj6\" (UID: \"fd170349-0424-48fa-a5e5-24781e3a21bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7tgj6" Nov 02 18:31:55.089567 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:55.089520 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd170349-0424-48fa-a5e5-24781e3a21bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-7tgj6\" (UID: \"fd170349-0424-48fa-a5e5-24781e3a21bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7tgj6" Nov 02 18:31:55.090179 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:55.090159 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd170349-0424-48fa-a5e5-24781e3a21bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-7tgj6\" (UID: \"fd170349-0424-48fa-a5e5-24781e3a21bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7tgj6" Nov 02 18:31:55.091788 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:55.091767 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd170349-0424-48fa-a5e5-24781e3a21bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-7tgj6\" (UID: \"fd170349-0424-48fa-a5e5-24781e3a21bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7tgj6" Nov 02 18:31:55.113758 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:55.113727 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jqw2\" (UniqueName: \"kubernetes.io/projected/fd170349-0424-48fa-a5e5-24781e3a21bb-kube-api-access-2jqw2\") pod \"maas-api-revoke-sub-f03629db-7tgj6\" (UID: \"fd170349-0424-48fa-a5e5-24781e3a21bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7tgj6" Nov 02 18:31:55.212131 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:55.212067 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7tgj6" Nov 02 18:31:55.507194 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:55.507161 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7tgj6"] Nov 02 18:31:55.507811 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:31:55.507784 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd170349_0424_48fa_a5e5_24781e3a21bb.slice/crio-98d1638b744f2a41764799b671aeaac46364b33845c44ceb02f1c071776f5cc4 WatchSource:0}: Error finding container 98d1638b744f2a41764799b671aeaac46364b33845c44ceb02f1c071776f5cc4: Status 404 returned error can't find the container with id 98d1638b744f2a41764799b671aeaac46364b33845c44ceb02f1c071776f5cc4 Nov 02 18:31:55.960263 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:55.960229 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7tgj6" event={"ID":"fd170349-0424-48fa-a5e5-24781e3a21bb","Type":"ContainerStarted","Data":"56cc8e37886df218f783bb3b373161fe761912b4c1e747d8239ccd03be496380"} Nov 02 18:31:55.960787 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:55.960282 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7tgj6" event={"ID":"fd170349-0424-48fa-a5e5-24781e3a21bb","Type":"ContainerStarted","Data":"98d1638b744f2a41764799b671aeaac46364b33845c44ceb02f1c071776f5cc4"} Nov 02 18:31:56.003118 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:56.003069 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7tgj6" podStartSLOduration=2.003055646 podStartE2EDuration="2.003055646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:31:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:31:56.001646524 +0000 UTC m=+3733.697001998" watchObservedRunningTime="2026-11-02 18:31:56.003055646 +0000 UTC m=+3733.698411118" Nov 02 18:31:56.965924 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:56.965887 4471 generic.go:356] "Generic (PLEG): container finished" podID="fd170349-0424-48fa-a5e5-24781e3a21bb" containerID="56cc8e37886df218f783bb3b373161fe761912b4c1e747d8239ccd03be496380" exitCode=0 Nov 02 18:31:56.966374 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:56.965974 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7tgj6" event={"ID":"fd170349-0424-48fa-a5e5-24781e3a21bb","Type":"ContainerDied","Data":"56cc8e37886df218f783bb3b373161fe761912b4c1e747d8239ccd03be496380"} Nov 02 18:31:58.105478 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:58.105447 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7tgj6" Nov 02 18:31:58.223883 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:58.223794 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd170349-0424-48fa-a5e5-24781e3a21bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd170349-0424-48fa-a5e5-24781e3a21bb-maas-api-openshift-ca\") pod \"fd170349-0424-48fa-a5e5-24781e3a21bb\" (UID: \"fd170349-0424-48fa-a5e5-24781e3a21bb\") " Nov 02 18:31:58.224046 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:58.223941 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd170349-0424-48fa-a5e5-24781e3a21bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd170349-0424-48fa-a5e5-24781e3a21bb-maas-api-xks-ca\") pod \"fd170349-0424-48fa-a5e5-24781e3a21bb\" (UID: \"fd170349-0424-48fa-a5e5-24781e3a21bb\") " Nov 02 18:31:58.224105 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:58.224051 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd170349-0424-48fa-a5e5-24781e3a21bb-kube-api-access-2jqw2\" (UniqueName: \"kubernetes.io/projected/fd170349-0424-48fa-a5e5-24781e3a21bb-kube-api-access-2jqw2\") pod \"fd170349-0424-48fa-a5e5-24781e3a21bb\" (UID: \"fd170349-0424-48fa-a5e5-24781e3a21bb\") " Nov 02 18:31:58.224169 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:58.224147 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd170349-0424-48fa-a5e5-24781e3a21bb-maas-api-openshift-ca" pod "fd170349-0424-48fa-a5e5-24781e3a21bb" (UID: "fd170349-0424-48fa-a5e5-24781e3a21bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:31:58.224373 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:58.224350 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd170349-0424-48fa-a5e5-24781e3a21bb-maas-api-openshift-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:31:58.226092 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:58.226064 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd170349-0424-48fa-a5e5-24781e3a21bb-maas-api-xks-ca" pod "fd170349-0424-48fa-a5e5-24781e3a21bb" (UID: "fd170349-0424-48fa-a5e5-24781e3a21bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:31:58.226215 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:58.226158 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd170349-0424-48fa-a5e5-24781e3a21bb-kube-api-access-2jqw2" pod "fd170349-0424-48fa-a5e5-24781e3a21bb" (UID: "fd170349-0424-48fa-a5e5-24781e3a21bb"). InnerVolumeSpecName "kube-api-access-2jqw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:31:58.325466 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:58.325433 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd170349-0424-48fa-a5e5-24781e3a21bb-maas-api-xks-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:31:58.325466 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:58.325462 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jqw2\" (UniqueName: \"kubernetes.io/projected/fd170349-0424-48fa-a5e5-24781e3a21bb-kube-api-access-2jqw2\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:31:58.976787 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:58.976753 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7tgj6" event={"ID":"fd170349-0424-48fa-a5e5-24781e3a21bb","Type":"ContainerDied","Data":"98d1638b744f2a41764799b671aeaac46364b33845c44ceb02f1c071776f5cc4"} Nov 02 18:31:58.976787 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:58.976784 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7tgj6" Nov 02 18:31:58.976787 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:31:58.976804 4471 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98d1638b744f2a41764799b671aeaac46364b33845c44ceb02f1c071776f5cc4" Nov 02 18:32:06.237225 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:06.237188 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:32:09.195036 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:32:09.194966 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:32:12.275786 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:12.275738 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4rln5"] Nov 02 18:32:12.276514 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:12.276489 4471 cpu_manager.go:414] "containerMap: removing container" podUID="fd170349-0424-48fa-a5e5-24781e3a21bb" containerName="revoke-keys" Nov 02 18:32:12.276581 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:12.276518 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd170349-0424-48fa-a5e5-24781e3a21bb" containerName="revoke-keys" Nov 02 18:32:12.276667 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:12.276654 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd170349-0424-48fa-a5e5-24781e3a21bb" containerName="revoke-keys" Nov 02 18:32:12.280726 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:12.280705 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4rln5" Nov 02 18:32:12.296264 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:12.296232 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:32:12.296264 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:12.296244 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 18:32:12.296502 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:12.296341 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 18:32:12.298367 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:12.298344 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4rln5"] Nov 02 18:32:12.308961 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:12.308935 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x9vwr\"" Nov 02 18:32:12.405280 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:12.405247 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3107ff10-c95a-4e3e-8e59-e0aa0a270b84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-4rln5\" (UID: \"3107ff10-c95a-4e3e-8e59-e0aa0a270b84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4rln5" Nov 02 18:32:12.405465 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:12.405376 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3107ff10-c95a-4e3e-8e59-e0aa0a270b84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-4rln5\" (UID: \"3107ff10-c95a-4e3e-8e59-e0aa0a270b84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4rln5" Nov 02 18:32:12.405465 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:12.405447 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j5dg\" (UniqueName: \"kubernetes.io/projected/3107ff10-c95a-4e3e-8e59-e0aa0a270b84-kube-api-access-8j5dg\") pod \"maas-api-revoke-sub-0165d7d2-4rln5\" (UID: \"3107ff10-c95a-4e3e-8e59-e0aa0a270b84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4rln5" Nov 02 18:32:12.506377 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:12.506343 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3107ff10-c95a-4e3e-8e59-e0aa0a270b84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-4rln5\" (UID: \"3107ff10-c95a-4e3e-8e59-e0aa0a270b84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4rln5" Nov 02 18:32:12.506588 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:12.506412 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j5dg\" (UniqueName: \"kubernetes.io/projected/3107ff10-c95a-4e3e-8e59-e0aa0a270b84-kube-api-access-8j5dg\") pod \"maas-api-revoke-sub-0165d7d2-4rln5\" (UID: \"3107ff10-c95a-4e3e-8e59-e0aa0a270b84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4rln5" Nov 02 18:32:12.506588 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:12.506521 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3107ff10-c95a-4e3e-8e59-e0aa0a270b84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-4rln5\" (UID: \"3107ff10-c95a-4e3e-8e59-e0aa0a270b84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4rln5" Nov 02 18:32:12.507035 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:12.507010 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3107ff10-c95a-4e3e-8e59-e0aa0a270b84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-4rln5\" (UID: \"3107ff10-c95a-4e3e-8e59-e0aa0a270b84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4rln5" Nov 02 18:32:12.508855 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:12.508832 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3107ff10-c95a-4e3e-8e59-e0aa0a270b84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-4rln5\" (UID: \"3107ff10-c95a-4e3e-8e59-e0aa0a270b84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4rln5" Nov 02 18:32:12.530529 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:12.530473 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j5dg\" (UniqueName: \"kubernetes.io/projected/3107ff10-c95a-4e3e-8e59-e0aa0a270b84-kube-api-access-8j5dg\") pod \"maas-api-revoke-sub-0165d7d2-4rln5\" (UID: \"3107ff10-c95a-4e3e-8e59-e0aa0a270b84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4rln5" Nov 02 18:32:12.594417 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:12.594387 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4rln5" Nov 02 18:32:12.887852 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:12.887796 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4rln5"] Nov 02 18:32:12.888096 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:32:12.888073 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3107ff10_c95a_4e3e_8e59_e0aa0a270b84.slice/crio-5b41de75b2268a388a7e9f4e9bb78403fcc104f6ba6707af4ba811c2c1ae3e43 WatchSource:0}: Error finding container 5b41de75b2268a388a7e9f4e9bb78403fcc104f6ba6707af4ba811c2c1ae3e43: Status 404 returned error can't find the container with id 5b41de75b2268a388a7e9f4e9bb78403fcc104f6ba6707af4ba811c2c1ae3e43 Nov 02 18:32:13.054179 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:13.054139 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4rln5" event={"ID":"3107ff10-c95a-4e3e-8e59-e0aa0a270b84","Type":"ContainerStarted","Data":"5b41de75b2268a388a7e9f4e9bb78403fcc104f6ba6707af4ba811c2c1ae3e43"} Nov 02 18:32:14.060950 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:14.060917 4471 generic.go:356] "Generic (PLEG): container finished" podID="3107ff10-c95a-4e3e-8e59-e0aa0a270b84" containerID="42b4cf29a6c07f86e4999dd48635889a6c6f0a9b0a63033751327588dc2c27e7" exitCode=0 Nov 02 18:32:14.061369 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:14.060955 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4rln5" event={"ID":"3107ff10-c95a-4e3e-8e59-e0aa0a270b84","Type":"ContainerDied","Data":"42b4cf29a6c07f86e4999dd48635889a6c6f0a9b0a63033751327588dc2c27e7"} Nov 02 18:32:15.207614 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:15.207584 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4rln5" Nov 02 18:32:15.335013 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:15.334921 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3107ff10-c95a-4e3e-8e59-e0aa0a270b84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3107ff10-c95a-4e3e-8e59-e0aa0a270b84-maas-api-openshift-ca\") pod \"3107ff10-c95a-4e3e-8e59-e0aa0a270b84\" (UID: \"3107ff10-c95a-4e3e-8e59-e0aa0a270b84\") " Nov 02 18:32:15.335013 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:15.335009 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3107ff10-c95a-4e3e-8e59-e0aa0a270b84-kube-api-access-8j5dg\" (UniqueName: \"kubernetes.io/projected/3107ff10-c95a-4e3e-8e59-e0aa0a270b84-kube-api-access-8j5dg\") pod \"3107ff10-c95a-4e3e-8e59-e0aa0a270b84\" (UID: \"3107ff10-c95a-4e3e-8e59-e0aa0a270b84\") " Nov 02 18:32:15.335363 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:15.335160 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3107ff10-c95a-4e3e-8e59-e0aa0a270b84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3107ff10-c95a-4e3e-8e59-e0aa0a270b84-maas-api-xks-ca\") pod \"3107ff10-c95a-4e3e-8e59-e0aa0a270b84\" (UID: \"3107ff10-c95a-4e3e-8e59-e0aa0a270b84\") " Nov 02 18:32:15.335442 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:15.335386 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3107ff10-c95a-4e3e-8e59-e0aa0a270b84-maas-api-openshift-ca" pod "3107ff10-c95a-4e3e-8e59-e0aa0a270b84" (UID: "3107ff10-c95a-4e3e-8e59-e0aa0a270b84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:32:15.335580 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:15.335562 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3107ff10-c95a-4e3e-8e59-e0aa0a270b84-maas-api-openshift-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:32:15.337152 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:15.337130 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3107ff10-c95a-4e3e-8e59-e0aa0a270b84-maas-api-xks-ca" pod "3107ff10-c95a-4e3e-8e59-e0aa0a270b84" (UID: "3107ff10-c95a-4e3e-8e59-e0aa0a270b84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:32:15.337152 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:15.337139 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3107ff10-c95a-4e3e-8e59-e0aa0a270b84-kube-api-access-8j5dg" pod "3107ff10-c95a-4e3e-8e59-e0aa0a270b84" (UID: "3107ff10-c95a-4e3e-8e59-e0aa0a270b84"). InnerVolumeSpecName "kube-api-access-8j5dg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:32:15.436937 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:15.436903 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j5dg\" (UniqueName: \"kubernetes.io/projected/3107ff10-c95a-4e3e-8e59-e0aa0a270b84-kube-api-access-8j5dg\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:32:15.436937 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:15.436931 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3107ff10-c95a-4e3e-8e59-e0aa0a270b84-maas-api-xks-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:32:16.071761 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:16.071733 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4rln5" Nov 02 18:32:16.071940 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:16.071760 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4rln5" event={"ID":"3107ff10-c95a-4e3e-8e59-e0aa0a270b84","Type":"ContainerDied","Data":"5b41de75b2268a388a7e9f4e9bb78403fcc104f6ba6707af4ba811c2c1ae3e43"} Nov 02 18:32:16.071940 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:16.071801 4471 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b41de75b2268a388a7e9f4e9bb78403fcc104f6ba6707af4ba811c2c1ae3e43" Nov 02 18:32:23.295512 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:23.295479 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wnmbb"] Nov 02 18:32:23.295941 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:23.295928 4471 cpu_manager.go:414] "containerMap: removing container" podUID="3107ff10-c95a-4e3e-8e59-e0aa0a270b84" containerName="revoke-keys" Nov 02 18:32:23.295988 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:23.295942 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3107ff10-c95a-4e3e-8e59-e0aa0a270b84" containerName="revoke-keys" Nov 02 18:32:23.296025 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:23.296007 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="3107ff10-c95a-4e3e-8e59-e0aa0a270b84" containerName="revoke-keys" Nov 02 18:32:23.299007 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:23.298988 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wnmbb" Nov 02 18:32:23.328829 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:23.328794 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:32:23.328998 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:23.328833 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 18:32:23.328998 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:23.328865 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 18:32:23.328998 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:23.328878 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x9vwr\"" Nov 02 18:32:23.332286 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:23.332261 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wnmbb"] Nov 02 18:32:23.411635 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:23.411593 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e70af26-9459-4cf4-8e1d-619383919641-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-wnmbb\" (UID: \"9e70af26-9459-4cf4-8e1d-619383919641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wnmbb" Nov 02 18:32:23.411635 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:23.411641 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tjwm\" (UniqueName: \"kubernetes.io/projected/9e70af26-9459-4cf4-8e1d-619383919641-kube-api-access-2tjwm\") pod \"maas-api-revoke-sub-6b19338f-wnmbb\" (UID: \"9e70af26-9459-4cf4-8e1d-619383919641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wnmbb" Nov 02 18:32:23.411849 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:23.411815 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e70af26-9459-4cf4-8e1d-619383919641-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-wnmbb\" (UID: \"9e70af26-9459-4cf4-8e1d-619383919641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wnmbb" Nov 02 18:32:23.512779 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:23.512744 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e70af26-9459-4cf4-8e1d-619383919641-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-wnmbb\" (UID: \"9e70af26-9459-4cf4-8e1d-619383919641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wnmbb" Nov 02 18:32:23.512935 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:23.512785 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tjwm\" (UniqueName: \"kubernetes.io/projected/9e70af26-9459-4cf4-8e1d-619383919641-kube-api-access-2tjwm\") pod \"maas-api-revoke-sub-6b19338f-wnmbb\" (UID: \"9e70af26-9459-4cf4-8e1d-619383919641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wnmbb" Nov 02 18:32:23.513170 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:23.513142 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e70af26-9459-4cf4-8e1d-619383919641-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-wnmbb\" (UID: \"9e70af26-9459-4cf4-8e1d-619383919641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wnmbb" Nov 02 18:32:23.513552 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:23.513524 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e70af26-9459-4cf4-8e1d-619383919641-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-wnmbb\" (UID: \"9e70af26-9459-4cf4-8e1d-619383919641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wnmbb" Nov 02 18:32:23.515631 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:23.515609 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e70af26-9459-4cf4-8e1d-619383919641-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-wnmbb\" (UID: \"9e70af26-9459-4cf4-8e1d-619383919641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wnmbb" Nov 02 18:32:23.536720 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:23.536698 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tjwm\" (UniqueName: \"kubernetes.io/projected/9e70af26-9459-4cf4-8e1d-619383919641-kube-api-access-2tjwm\") pod \"maas-api-revoke-sub-6b19338f-wnmbb\" (UID: \"9e70af26-9459-4cf4-8e1d-619383919641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wnmbb" Nov 02 18:32:23.613453 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:23.613376 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wnmbb" Nov 02 18:32:23.906417 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:23.906386 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wnmbb"] Nov 02 18:32:23.906568 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:32:23.906449 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e70af26_9459_4cf4_8e1d_619383919641.slice/crio-ca414371dccdaaacf4b732e951e438be870e43f7c040c4e85c65f99a135d7321 WatchSource:0}: Error finding container ca414371dccdaaacf4b732e951e438be870e43f7c040c4e85c65f99a135d7321: Status 404 returned error can't find the container with id ca414371dccdaaacf4b732e951e438be870e43f7c040c4e85c65f99a135d7321 Nov 02 18:32:24.115771 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:24.115735 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wnmbb" event={"ID":"9e70af26-9459-4cf4-8e1d-619383919641","Type":"ContainerStarted","Data":"ca414371dccdaaacf4b732e951e438be870e43f7c040c4e85c65f99a135d7321"} Nov 02 18:32:25.122301 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:25.122268 4471 generic.go:356] "Generic (PLEG): container finished" podID="9e70af26-9459-4cf4-8e1d-619383919641" containerID="fb0818761bf84c1271a65d5679570cabeb33d32c823e65194292a8fe27726f5f" exitCode=0 Nov 02 18:32:25.122719 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:25.122357 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wnmbb" event={"ID":"9e70af26-9459-4cf4-8e1d-619383919641","Type":"ContainerDied","Data":"fb0818761bf84c1271a65d5679570cabeb33d32c823e65194292a8fe27726f5f"} Nov 02 18:32:26.272041 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:26.272011 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wnmbb" Nov 02 18:32:26.438826 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:26.438735 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e70af26-9459-4cf4-8e1d-619383919641-kube-api-access-2tjwm\" (UniqueName: \"kubernetes.io/projected/9e70af26-9459-4cf4-8e1d-619383919641-kube-api-access-2tjwm\") pod \"9e70af26-9459-4cf4-8e1d-619383919641\" (UID: \"9e70af26-9459-4cf4-8e1d-619383919641\") " Nov 02 18:32:26.438826 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:26.438803 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e70af26-9459-4cf4-8e1d-619383919641-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e70af26-9459-4cf4-8e1d-619383919641-maas-api-xks-ca\") pod \"9e70af26-9459-4cf4-8e1d-619383919641\" (UID: \"9e70af26-9459-4cf4-8e1d-619383919641\") " Nov 02 18:32:26.439033 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:26.438860 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e70af26-9459-4cf4-8e1d-619383919641-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e70af26-9459-4cf4-8e1d-619383919641-maas-api-openshift-ca\") pod \"9e70af26-9459-4cf4-8e1d-619383919641\" (UID: \"9e70af26-9459-4cf4-8e1d-619383919641\") " Nov 02 18:32:26.439277 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:26.439251 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e70af26-9459-4cf4-8e1d-619383919641-maas-api-openshift-ca" pod "9e70af26-9459-4cf4-8e1d-619383919641" (UID: "9e70af26-9459-4cf4-8e1d-619383919641"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:32:26.440917 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:26.440891 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e70af26-9459-4cf4-8e1d-619383919641-maas-api-xks-ca" pod "9e70af26-9459-4cf4-8e1d-619383919641" (UID: "9e70af26-9459-4cf4-8e1d-619383919641"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:32:26.441017 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:26.440942 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e70af26-9459-4cf4-8e1d-619383919641-kube-api-access-2tjwm" pod "9e70af26-9459-4cf4-8e1d-619383919641" (UID: "9e70af26-9459-4cf4-8e1d-619383919641"). InnerVolumeSpecName "kube-api-access-2tjwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:32:26.540626 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:26.540595 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tjwm\" (UniqueName: \"kubernetes.io/projected/9e70af26-9459-4cf4-8e1d-619383919641-kube-api-access-2tjwm\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:32:26.540626 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:26.540621 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e70af26-9459-4cf4-8e1d-619383919641-maas-api-xks-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:32:26.540626 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:26.540632 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e70af26-9459-4cf4-8e1d-619383919641-maas-api-openshift-ca\") on node \"ip-10-0-142-26.ec2.internal\" DevicePath \"\"" Nov 02 18:32:27.132365 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:27.132330 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wnmbb" Nov 02 18:32:27.132570 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:27.132336 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wnmbb" event={"ID":"9e70af26-9459-4cf4-8e1d-619383919641","Type":"ContainerDied","Data":"ca414371dccdaaacf4b732e951e438be870e43f7c040c4e85c65f99a135d7321"} Nov 02 18:32:27.132570 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:27.132469 4471 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca414371dccdaaacf4b732e951e438be870e43f7c040c4e85c65f99a135d7321" Nov 02 18:32:34.637007 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:34.636969 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f6b9c"] Nov 02 18:32:39.182351 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:32:39.182295 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:32:45.680347 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:45.680282 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5677bb6689-t5vq2_8666c0bd-1a97-4bb3-8695-8f84b7047250/manager/0.log" Nov 02 18:32:46.189925 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:46.189882 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-8hfs4_7092e32e-80ce-47db-acfc-57735ad62658/manager/0.log" Nov 02 18:32:46.709238 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:46.709208 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67b9f44df6-zft8x_eae8886c-0696-40df-b91f-2fbd5b50e65e/manager/0.log" Nov 02 18:32:48.589218 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:48.589185 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-4rln5_3107ff10-c95a-4e3e-8e59-e0aa0a270b84/revoke-keys/0.log" Nov 02 18:32:49.355639 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:49.355610 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-wnmbb_9e70af26-9459-4cf4-8e1d-619383919641/revoke-keys/0.log" Nov 02 18:32:49.540096 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:49.540065 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-7tgj6_fd170349-0424-48fa-a5e5-24781e3a21bb/revoke-keys/0.log" Nov 02 18:32:52.359772 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:52.359738 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-f6b9c_ab7e11d6-997a-4765-a1b7-85daa191d013/limitador/0.log" Nov 02 18:32:53.553184 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:53.553140 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-sxvjv_43917fff-d5ed-4c3f-b4e4-b74d98f39e21/discovery/0.log" Nov 02 18:32:53.765455 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:53.765416 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78648788bf-gnws5_7061bad7-2417-4dc7-a025-b1fcb9aec58d/kube-auth-proxy/0.log" Nov 02 18:32:54.357573 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:54.357541 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-5hhwn_a482b42d-8782-433f-a05f-391d4516a850/istio-proxy/0.log" Nov 02 18:32:54.955516 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:54.955482 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-cv4r2_6c0a1b02-6f85-4e92-be26-a9f0cb390cef/istio-proxy/0.log" Nov 02 18:32:55.155656 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:55.155617 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-f6kf2_c40d5295-d354-4be9-89b8-6bd96ae2e056/istio-proxy/0.log" Nov 02 18:32:55.354492 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:55.354460 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-k97sr_ef3e0f41-024e-4c23-80c4-5cc5b5d08331/istio-proxy/0.log" Nov 02 18:32:56.347705 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:56.347674 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-2p9rr_85011b6f-dbf8-472b-bf28-f13cc9d99de4/payload-pre-processing/0.log" Nov 02 18:32:56.539646 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:56.539617 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rjwjd_91ff9502-0ff4-43a1-8f78-b5d8c8fe683e/payload-processing/0.log" Nov 02 18:32:57.770194 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:57.770163 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-drrc4_880a572e-4d2f-4f0c-9f31-1f40f37c1723/main/0.log" Nov 02 18:32:57.963003 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:57.962973 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-v25sf_ad9d1c12-a624-47ee-80b2-db72091ec1f6/main/0.log" Nov 02 18:32:58.366143 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:58.366109 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66_26722ed8-2267-4729-8872-7fbf9a222c87/main/0.log" Nov 02 18:32:58.566417 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:58.566387 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-j2l7j_b966a58b-3f29-4f26-a84e-89999a77de25/main/0.log" Nov 02 18:32:58.942860 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:32:58.942828 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h_5c5408ad-4b7b-47c0-854a-6e6c3a723302/main/0.log" Nov 02 18:33:09.124852 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:33:09.124815 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5677bb6689-t5vq2_8666c0bd-1a97-4bb3-8695-8f84b7047250/manager/0.log" Nov 02 18:33:09.182133 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:33:09.182091 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:33:09.556604 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:33:09.556573 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-8hfs4_7092e32e-80ce-47db-acfc-57735ad62658/manager/0.log" Nov 02 18:33:09.994852 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:33:09.994771 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67b9f44df6-zft8x_eae8886c-0696-40df-b91f-2fbd5b50e65e/manager/0.log" Nov 02 18:33:11.581017 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:33:11.580988 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-4rln5_3107ff10-c95a-4e3e-8e59-e0aa0a270b84/revoke-keys/0.log" Nov 02 18:33:12.169132 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:33:12.169101 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-wnmbb_9e70af26-9459-4cf4-8e1d-619383919641/revoke-keys/0.log" Nov 02 18:33:12.351755 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:33:12.351727 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-7tgj6_fd170349-0424-48fa-a5e5-24781e3a21bb/revoke-keys/0.log" Nov 02 18:33:14.732029 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:33:14.731996 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-f6b9c_ab7e11d6-997a-4765-a1b7-85daa191d013/limitador/0.log" Nov 02 18:33:15.696298 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:33:15.696259 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-sxvjv_43917fff-d5ed-4c3f-b4e4-b74d98f39e21/discovery/0.log" Nov 02 18:33:15.901410 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:33:15.901378 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78648788bf-gnws5_7061bad7-2417-4dc7-a025-b1fcb9aec58d/kube-auth-proxy/0.log" Nov 02 18:33:16.506166 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:33:16.506131 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-5hhwn_a482b42d-8782-433f-a05f-391d4516a850/istio-proxy/0.log" Nov 02 18:33:17.077214 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:33:17.077179 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-cv4r2_6c0a1b02-6f85-4e92-be26-a9f0cb390cef/istio-proxy/0.log" Nov 02 18:33:17.272457 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:33:17.272420 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-f6kf2_c40d5295-d354-4be9-89b8-6bd96ae2e056/istio-proxy/0.log" Nov 02 18:33:17.460611 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:33:17.460525 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-k97sr_ef3e0f41-024e-4c23-80c4-5cc5b5d08331/istio-proxy/0.log" Nov 02 18:33:18.456319 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:33:18.456267 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-2p9rr_85011b6f-dbf8-472b-bf28-f13cc9d99de4/payload-pre-processing/0.log" Nov 02 18:33:18.628651 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:33:18.628614 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rjwjd_91ff9502-0ff4-43a1-8f78-b5d8c8fe683e/payload-processing/0.log" Nov 02 18:33:19.553317 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:33:19.553271 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-drrc4_880a572e-4d2f-4f0c-9f31-1f40f37c1723/main/0.log" Nov 02 18:33:19.745381 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:33:19.745346 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-v25sf_ad9d1c12-a624-47ee-80b2-db72091ec1f6/main/0.log" Nov 02 18:33:20.146013 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:33:20.145977 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jrc66_26722ed8-2267-4729-8872-7fbf9a222c87/main/0.log" Nov 02 18:33:20.341042 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:33:20.341011 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-j2l7j_b966a58b-3f29-4f26-a84e-89999a77de25/main/0.log" Nov 02 18:33:20.729840 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:33:20.729807 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-hpk4h_5c5408ad-4b7b-47c0-854a-6e6c3a723302/main/0.log" Nov 02 18:33:39.182549 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:33:39.182516 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:33:55.036692 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:33:55.036654 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67b9f44df6-zft8x_eae8886c-0696-40df-b91f-2fbd5b50e65e/manager/0.log" Nov 02 18:33:55.321819 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:33:55.321741 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67b9f44df6-zft8x_eae8886c-0696-40df-b91f-2fbd5b50e65e/manager/0.log" Nov 02 18:33:55.984174 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:33:55.984144 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rjwjd_91ff9502-0ff4-43a1-8f78-b5d8c8fe683e/payload-processing/0.log" Nov 02 18:33:56.157140 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:33:56.157110 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rjwjd_91ff9502-0ff4-43a1-8f78-b5d8c8fe683e/payload-processing/0.log" Nov 02 18:34:09.182771 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:34:09.182731 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:34:38.436843 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:38.436807 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67b9f44df6-zft8x_eae8886c-0696-40df-b91f-2fbd5b50e65e/manager/0.log" Nov 02 18:34:38.793708 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:38.793670 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67b9f44df6-zft8x_eae8886c-0696-40df-b91f-2fbd5b50e65e/manager/0.log" Nov 02 18:34:39.182819 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:34:39.182731 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:34:39.468730 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:39.468654 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rjwjd_91ff9502-0ff4-43a1-8f78-b5d8c8fe683e/payload-processing/0.log" Nov 02 18:34:39.642457 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:39.642428 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rjwjd_91ff9502-0ff4-43a1-8f78-b5d8c8fe683e/payload-processing/0.log" Nov 02 18:34:44.870501 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:44.870466 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-l2q87_83c827e3-5338-49b5-97a1-56bdb5904500/global-pull-secret-syncer/0.log" Nov 02 18:34:46.063848 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:46.063817 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rt5cp_1b5ee10f-87a1-4168-904e-ba233f304fe3/konnectivity-agent/0.log" Nov 02 18:34:46.686089 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:46.686052 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-26.ec2.internal_1b001a1b5ad5b960c3a58390390fe0a6/haproxy/0.log" Nov 02 18:34:53.213816 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:53.213784 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-f6b9c_ab7e11d6-997a-4765-a1b7-85daa191d013/limitador/0.log" Nov 02 18:34:54.840403 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:54.840283 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_00b0092d-4e1f-4bb5-97a3-4e6e7e369d82/alertmanager/0.log" Nov 02 18:34:54.959472 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:54.959440 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_00b0092d-4e1f-4bb5-97a3-4e6e7e369d82/config-reloader/0.log" Nov 02 18:34:55.032093 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.032064 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dt9rz/perf-node-gather-daemonset-6nsf5"] Nov 02 18:34:55.032657 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.032636 4471 cpu_manager.go:414] "containerMap: removing container" podUID="9e70af26-9459-4cf4-8e1d-619383919641" containerName="revoke-keys" Nov 02 18:34:55.032657 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.032658 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e70af26-9459-4cf4-8e1d-619383919641" containerName="revoke-keys" Nov 02 18:34:55.032801 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.032737 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e70af26-9459-4cf4-8e1d-619383919641" containerName="revoke-keys" Nov 02 18:34:55.035814 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.035795 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-6nsf5" Nov 02 18:34:55.052451 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.052424 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dt9rz\"/\"kube-root-ca.crt\"" Nov 02 18:34:55.052562 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.052538 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dt9rz\"/\"openshift-service-ca.crt\"" Nov 02 18:34:55.054247 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.054223 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dt9rz/perf-node-gather-daemonset-6nsf5"] Nov 02 18:34:55.064606 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.064584 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dt9rz\"/\"default-dockercfg-rxf8r\"" Nov 02 18:34:55.081729 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.081708 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_00b0092d-4e1f-4bb5-97a3-4e6e7e369d82/kube-rbac-proxy-web/0.log" Nov 02 18:34:55.095190 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.095137 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/365df340-6161-4f84-8aef-63322405a94a-lib-modules\") pod \"perf-node-gather-daemonset-6nsf5\" (UID: \"365df340-6161-4f84-8aef-63322405a94a\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-6nsf5" Nov 02 18:34:55.095190 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.095168 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/365df340-6161-4f84-8aef-63322405a94a-proc\") pod \"perf-node-gather-daemonset-6nsf5\" (UID: \"365df340-6161-4f84-8aef-63322405a94a\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-6nsf5" Nov 02 18:34:55.095386 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.095193 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/365df340-6161-4f84-8aef-63322405a94a-podres\") pod \"perf-node-gather-daemonset-6nsf5\" (UID: \"365df340-6161-4f84-8aef-63322405a94a\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-6nsf5" Nov 02 18:34:55.095386 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.095290 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p98hv\" (UniqueName: \"kubernetes.io/projected/365df340-6161-4f84-8aef-63322405a94a-kube-api-access-p98hv\") pod \"perf-node-gather-daemonset-6nsf5\" (UID: \"365df340-6161-4f84-8aef-63322405a94a\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-6nsf5" Nov 02 18:34:55.095472 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.095424 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/365df340-6161-4f84-8aef-63322405a94a-sys\") pod \"perf-node-gather-daemonset-6nsf5\" (UID: \"365df340-6161-4f84-8aef-63322405a94a\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-6nsf5" Nov 02 18:34:55.196667 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.196633 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/365df340-6161-4f84-8aef-63322405a94a-sys\") pod \"perf-node-gather-daemonset-6nsf5\" (UID: \"365df340-6161-4f84-8aef-63322405a94a\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-6nsf5" Nov 02 18:34:55.196800 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.196753 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/365df340-6161-4f84-8aef-63322405a94a-sys\") pod \"perf-node-gather-daemonset-6nsf5\" (UID: \"365df340-6161-4f84-8aef-63322405a94a\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-6nsf5" Nov 02 18:34:55.196858 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.196792 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/365df340-6161-4f84-8aef-63322405a94a-lib-modules\") pod \"perf-node-gather-daemonset-6nsf5\" (UID: \"365df340-6161-4f84-8aef-63322405a94a\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-6nsf5" Nov 02 18:34:55.196858 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.196828 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/365df340-6161-4f84-8aef-63322405a94a-proc\") pod \"perf-node-gather-daemonset-6nsf5\" (UID: \"365df340-6161-4f84-8aef-63322405a94a\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-6nsf5" Nov 02 18:34:55.196969 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.196862 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/365df340-6161-4f84-8aef-63322405a94a-podres\") pod \"perf-node-gather-daemonset-6nsf5\" (UID: \"365df340-6161-4f84-8aef-63322405a94a\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-6nsf5" Nov 02 18:34:55.196969 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.196887 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/365df340-6161-4f84-8aef-63322405a94a-lib-modules\") pod \"perf-node-gather-daemonset-6nsf5\" (UID: \"365df340-6161-4f84-8aef-63322405a94a\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-6nsf5" Nov 02 18:34:55.196969 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.196891 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p98hv\" (UniqueName: \"kubernetes.io/projected/365df340-6161-4f84-8aef-63322405a94a-kube-api-access-p98hv\") pod \"perf-node-gather-daemonset-6nsf5\" (UID: \"365df340-6161-4f84-8aef-63322405a94a\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-6nsf5" Nov 02 18:34:55.197092 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.196973 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/365df340-6161-4f84-8aef-63322405a94a-proc\") pod \"perf-node-gather-daemonset-6nsf5\" (UID: \"365df340-6161-4f84-8aef-63322405a94a\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-6nsf5" Nov 02 18:34:55.197092 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.196985 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/365df340-6161-4f84-8aef-63322405a94a-podres\") pod \"perf-node-gather-daemonset-6nsf5\" (UID: \"365df340-6161-4f84-8aef-63322405a94a\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-6nsf5" Nov 02 18:34:55.201934 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.201910 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_00b0092d-4e1f-4bb5-97a3-4e6e7e369d82/kube-rbac-proxy/0.log" Nov 02 18:34:55.223400 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.223377 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p98hv\" (UniqueName: \"kubernetes.io/projected/365df340-6161-4f84-8aef-63322405a94a-kube-api-access-p98hv\") pod \"perf-node-gather-daemonset-6nsf5\" (UID: \"365df340-6161-4f84-8aef-63322405a94a\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-6nsf5" Nov 02 18:34:55.321430 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.321406 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_00b0092d-4e1f-4bb5-97a3-4e6e7e369d82/kube-rbac-proxy-metric/0.log" Nov 02 18:34:55.349276 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.349225 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-6nsf5" Nov 02 18:34:55.441961 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.441935 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_00b0092d-4e1f-4bb5-97a3-4e6e7e369d82/prom-label-proxy/0.log" Nov 02 18:34:55.564469 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.564442 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_00b0092d-4e1f-4bb5-97a3-4e6e7e369d82/init-config-reloader/0.log" Nov 02 18:34:55.644166 ip-10-0-142-26 kubenswrapper[4471]: W1102 18:34:55.644136 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod365df340_6161_4f84_8aef_63322405a94a.slice/crio-02fd4bb70c7d517ab43722e83e6a8fe09bab4b107fdc18a5efcc611f319e63ff WatchSource:0}: Error finding container 02fd4bb70c7d517ab43722e83e6a8fe09bab4b107fdc18a5efcc611f319e63ff: Status 404 returned error can't find the container with id 02fd4bb70c7d517ab43722e83e6a8fe09bab4b107fdc18a5efcc611f319e63ff Nov 02 18:34:55.645066 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.645036 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dt9rz/perf-node-gather-daemonset-6nsf5"] Nov 02 18:34:55.845821 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.845796 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-6hfdn_6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c/kube-state-metrics/0.log" Nov 02 18:34:55.847712 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.847688 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-6nsf5" event={"ID":"365df340-6161-4f84-8aef-63322405a94a","Type":"ContainerStarted","Data":"a06da342e1a396a6e2481dfddb98bbb58d3a39933b1a71fdbfe1fac732a22863"} Nov 02 18:34:55.847821 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.847747 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-6nsf5" Nov 02 18:34:55.847821 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.847769 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-6nsf5" event={"ID":"365df340-6161-4f84-8aef-63322405a94a","Type":"ContainerStarted","Data":"02fd4bb70c7d517ab43722e83e6a8fe09bab4b107fdc18a5efcc611f319e63ff"} Nov 02 18:34:55.890039 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.889990 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-6nsf5" podStartSLOduration=0.88997836 podStartE2EDuration="889.97836ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 18:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:34:55.888516683 +0000 UTC m=+3913.583872155" watchObservedRunningTime="2026-11-02 18:34:55.88997836 +0000 UTC m=+3913.585333832" Nov 02 18:34:55.957394 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:55.957330 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-6hfdn_6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c/kube-rbac-proxy-main/0.log" Nov 02 18:34:56.076769 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:56.076744 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-6hfdn_6fd6596b-f5c4-4ad5-b5d8-79be4df1b56c/kube-rbac-proxy-self/0.log" Nov 02 18:34:56.297358 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:56.297329 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-t75j5_f5525bdf-02c9-4225-b7a2-ae353e0ba168/monitoring-plugin/0.log" Nov 02 18:34:56.788257 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:56.788231 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v5w55_219e320d-1e5e-4c4f-9eaf-5ee7411104b0/node-exporter/0.log" Nov 02 18:34:56.905854 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:56.905821 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v5w55_219e320d-1e5e-4c4f-9eaf-5ee7411104b0/kube-rbac-proxy/0.log" Nov 02 18:34:57.030286 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:57.030261 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v5w55_219e320d-1e5e-4c4f-9eaf-5ee7411104b0/init-textfile/0.log" Nov 02 18:34:58.737182 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:58.737139 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rcjnj_1cfc0b3c-a7f2-4cae-99ba-23286bfb0cc1/prometheus-operator/0.log" Nov 02 18:34:58.860700 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:34:58.860669 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rcjnj_1cfc0b3c-a7f2-4cae-99ba-23286bfb0cc1/kube-rbac-proxy/0.log" Nov 02 18:35:01.863387 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:35:01.863359 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-6nsf5" Nov 02 18:35:03.413524 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:35:03.413492 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-56665b75dc-whm5m_cde2c4ed-f0d0-4c35-8b75-8a8117bfb819/console/0.log" Nov 02 18:35:03.548197 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:35:03.548165 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-st8b5_4fd49e19-2141-451a-9c0b-2616cf2d6c71/download-server/0.log" Nov 02 18:35:07.093380 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:35:07.093349 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bkdn9_4d601642-0fa6-4b68-b9cf-324c59e41637/dns/0.log" Nov 02 18:35:07.207601 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:35:07.207570 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bkdn9_4d601642-0fa6-4b68-b9cf-324c59e41637/kube-rbac-proxy/0.log" Nov 02 18:35:07.821427 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:35:07.821387 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-22fc9_665ac753-596e-4d13-b8c1-7a1c7e791cf8/dns-node-resolver/0.log" Nov 02 18:35:09.189901 ip-10-0-142-26 kubenswrapper[4471]: E1102 18:35:09.189868 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ccwcv" containerName="extractor" Nov 02 18:35:09.333675 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:35:09.333611 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bz42x_0aed3a0d-f6c0-4955-8d67-8b8b98bebe25/node-ca/0.log" Nov 02 18:35:12.972439 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:35:12.972386 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-sxvjv_43917fff-d5ed-4c3f-b4e4-b74d98f39e21/discovery/0.log" Nov 02 18:35:13.007048 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:35:13.007015 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78648788bf-gnws5_7061bad7-2417-4dc7-a025-b1fcb9aec58d/kube-auth-proxy/0.log" Nov 02 18:35:13.200550 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:35:13.200502 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-5hhwn_a482b42d-8782-433f-a05f-391d4516a850/istio-proxy/0.log" Nov 02 18:35:13.460959 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:35:13.460918 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-cv4r2_6c0a1b02-6f85-4e92-be26-a9f0cb390cef/istio-proxy/0.log" Nov 02 18:35:13.528097 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:35:13.528058 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-f6kf2_c40d5295-d354-4be9-89b8-6bd96ae2e056/istio-proxy/0.log" Nov 02 18:35:13.595183 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:35:13.595154 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-k97sr_ef3e0f41-024e-4c23-80c4-5cc5b5d08331/istio-proxy/0.log" Nov 02 18:35:13.909655 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:35:13.909621 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-2p9rr_85011b6f-dbf8-472b-bf28-f13cc9d99de4/payload-pre-processing/0.log" Nov 02 18:35:13.975582 ip-10-0-142-26 kubenswrapper[4471]: I1102 18:35:13.975553 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rjwjd_91ff9502-0ff4-43a1-8f78-b5d8c8fe683e/payload-processing/0.log"